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PROLOGUE 


It  is  3:00  pm  and  the  exercise  has  just  ended.  The  Director  of  Training  and 
Education  reminds  all  the  participants  that  an  After  Action  Review  (AAR)  still  needs  to 
be  done  and  she  promises  to  get  everyone  out  of  the  office  by  4:30.  Tired  response 
partners  are  quickly  gathered  into  a  windowless  conference  room  where  large  round  gray 
tables  are  sprinkled  with  red  and  green  Jolly  Rancher  hard  candies.  Every  table  has  a  sign 
that  specifies  an  incident  command  section  and  each  individual  is  instructed  to  be  seated 
at  the  table  that  corresponds  to  the  sectional  roles  they  played  during  the  exercise.  A 
murmur  of  anticipation  is  quieted  down  by  the  voice  of  the  training  director  who  will  act 
as  the  session’s  facilitator.  As  custom,  all  are  told  that  the  purpose  of  the  session  is  to 
leam  from  the  exercise  so  that  performance  can  be  improved  during  a  real  emergency. 

After  a  few  latecomers  arrive  and  are  seated,  the  facilitator  proceeds  to  make  it 
clear  that  the  session  should  not  be  used  to  lay  blame,  tear  anyone  down  or  point  fingers. 
Individuals  are  encouraged  to  be  positive,  open,  and  to  speak  with  candor  and  integrity. 
The  focus  should  be  on  learning  and  improving  future  performance.  The  group  is  told 
that  everything  said  during  the  process  will  be  held  in  confidence  and  that  what  is  shared 
will  stay  “in-house.”  All  participants  have  been  encouraged  to  engage  seriously  because 
everyone  has  something  important  to  share  about  what  has  happened  during  the  exercise. 

Reassuringly,  it  is  also  made  clear  that  an  AAR  report  will  be  developed  to 
capture  all  “lessons  learned”  and  that  the  document  will  be  uploaded  to  an  in-house 
document  sharing  system  for  all  to  review  once  it  is  complete.  It  is  understood  that  this 
document  will  also  influence  the  updating  of  all  pertinent  plans,  procedures,  and 
capabilities.  Finally,  the  facilitator  begins  the  process  by  asking,  “OK,  let’s  start  the 
AAR.  Is  everyone  ready?”  With  that  cue,  most  in  the  room  look  up  at  the  clock,  carefully 
peel  the  wrapper  off  a  tacky  Jolly  Rancher  and  prepare  to  share  their  experiences  of  that 
afternoon’s  exercise. 
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I.  INTRODUCTION 


A.  ARE  WE  LEARNING  YET? 

It  is  a  good  possibility  that  most  emergency  response  professionals  within  the 
homeland  security  enterprise,  especially  since  the  events  of  9/11,  have  encountered  a 
similar  situation  to  the  one  narrated  above.  However,  how  is  it  possible  to  know  that 
response  professionals  (or  their  organizations)  have  actually  learned  something  during  or 
from  the  exercise  or  the  subsequent  learning  events?  Even  if  they  have  learned 
something,  what  guarantee  is  there  that  their  behaviors  or  attitudes  will  actually  change  in 
the  future  because  of  this  new  knowledge? 

The  answers  to  these  questions  are  important.  The  United  States  of  America  has 
made  a  significant  investment  in  the  security  of  this  country.  In  a  recent  article,  Mueller 
and  Stewart  report,  “the  increase  in  expenditures  on  domestic  homeland  security  over  the 
decade  exceeds  one  trillion  dollars.”  In  the  same  article  they  ask,  “are  the  gains  in 
security  worth  the  funds  expended?”1  This  researcher  believes  a  corollary  to  that  question 
is  what  knowledge  has  been  gained  from  these  expenditures  and  how  is  it  being  used? 
The  beginnings  of  an  answer  may  be  at  hand;  unfortunately,  it  may  not  be  what  was 
desired. 

It  is  becoming  clear  that  within  many  agencies,  this  staple  of  post-exercise 
activity  is  not  being  practiced  regularly  and  that  the  lessons  learned  from  the  completed 
AARs  are  not  distributed  or  implemented  effectively.2  If  agencies  are  not  even  taking  the 
time  to  elicit  and  capture  the  knowledge  that  might  be  gleaned  from  an  AAR,  they  will 
most  certainly  have  trouble  transferring  and  applying  this  knowledge  in  the  future.  In 
other  words,  the  AAR  may  not  be  reliably  and  effectively  serving  its  intended  purpose. 


1  John  Mueller  and  Mark  G.  Stewart,  “Terror,  Security  and  Money:  Balancing  the  Risks  Benefits  and 
Costs  of  Homeland  Security,”  Presented  at  the  Annual  of  the  Midwest  Political  Science  Association, 
Chicago,  IL,  April  1,  2011,  2. 

2  Department  of  Homeland  Security,  Office  of  the  Inspector  General,  “FEMA’s  Progress  in 
Implementing  the  Remedial  Action  Management  Program,”  OIG-1 1-32,  January  2011. 
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These  missed  “learning  opportunities”  not  only  reflect  the  loss  of  large  sums  of  money 
due  to  an  inability  to  learn  from  the  past,  they  also  reflect  the  potential  loss  of  life  and 
limb  in  the  future. 

B.  STATEMENT  OF  THE  PROBLEM 

The  AAR  is  now  commonly  practiced  after  most  emergency  exercises  and  events 
in  the  hopes  of  detennining  how  well  objectives  have  been  met  so  lessons  can  be  learned 
and  perfonnance  can  be  improved  during  real  emergencies.  Moynihan  refers  to  this  type 
of  learning  as  “intercrisis  learning,”  as  opposed  to  “intracrisis  learning,”  which  occurs 
during  an  emergency. 3  Ultimately,  the  AAR  is  a  vehicle  intended  to  promote  and  increase 
individual  and  organizational  learning.  However,  over  the  last  few  years,  the  AAR  has 
come  under  fire  within  the  homeland  security  enterprise. 

One  serious  problem  is  that  researchers  are  unable  to  answer  basic  questions 
about  how  individuals  leam  during  exercises  or  the  actual  the  AAR  process.  Donahue  and 
Tuohy  address  this  issue  by  stating: 

Despite  these  widespread  activities,  however,  the  term  “lessons  learned”  is 
often  a  misnomer.  Our  experience  suggests  that  purported  lessons  learned 
are  not  really  learned',  many  problems  and  mistakes  are  repeated  in 
subsequent  events.  It  appears  that  while  review  of  incidents  and  the 
identification  of  lessons  are  more  readily  accomplished,  true  learning  is 
much  more  difficult.  Reports  and  lessons  are  often  ignored,  and  even 
when  they  are  not,  lessons  are  too  often  isolated  and  perishable,  rather 
than  generalized  and  institutionalized.4 

Why  are  these  lessons  “not  really  learned”?  Birkland  has  come  up  with  a  fairly 
logical  and  pragmatic  explanation  to  the  problem: 

It  is  difficult  to  claim  that  any  actual  learning  occurred  because 
insufficient  time  has  elapsed  between  the  event,  the  creation  of  the  report, 
and  any  subsequent  tests  of  the  “lessons.”  Instead,  these  documents  really 


3  Donald  P.  Moynihan,  “From  Intercrisis  to  Intracrisis  Learning,”  Journal  of  Contingencies  and  Crisis 
Management  17,  no.  3  (September  2009):  189. 

4  Amy  Donahue  and  Robert  Tuohy,  “Lessons  We  Don’t  Learn:  A  Study  of  the  Lessons  of  Disasters, 
Why  We  Repeat  Them,  and  Flow  We  Can  Learn  Them,”  Homeland  Security’  Affairs  II,  no.  2  (July  2006):  3, 
http  ://www.hsaj.org/?article=2.2.4. 
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focus  on  “lessons  observed”  or,  more  simply,  the  observations  that 
officials  and  experts  made  about  the  preparations  before  and  responses  to 
the  crisis  or  disaster.5 

In  essence,  a  lesson  is  not  really  learned  until  such  time  in  the  future  that  evidence  exists 
that  knowledge,  or  more  importantly,  behavior  has  changed  because  of  the  process.  In 
other  words,  until  such  time  that  a  lesson  can  be  tested;  it  is  unknown  as  to  whether  or  not 
anything  has  been  truly  learned. 

Another  more  foundational  problem  may  be  that  although  the  actual  AAR  process 
was  developed  almost  40  years  ago,  not  much  is  known  about  how  it  actually  works  at  a 
theoretical  level.  The  practice  is  not  well  grounded  in  the  learning  literature.  In  fact, 
much  of  the  literature  that  does  exist  on  the  topic  relies  on  untested  claims  or  conjecture 
while  scientific  research  on  related  and  better  researched  organizational  learning  topics, 
such  as  single  and  double  loop  learning,  is  thin  or  non-existent. 

It  is  also  important  to  note  that  AAR  facilitators  may  have  varying  levels  of 
expertise  when  it  comes  to  directing  the  process.  Mastaglio  et  al.  state  that,  “Different 
facilitators  may  be  more  or  less  effective  at  stimulating  discussion,  leading  participants, 
structuring  the  lessons,  reflecting  on  planned  goals,  and  synthesizing  the  lessons  learned 
from  the  AAR.”6  The  authors  also  state  that,  “These  issues  potentially  cause  the  AAR  to 
be  highly  important  but  inconsistent  and  their  outcomes  can  therefore  be  unpredictable.”7 

Also,  because  an  AAR  does  not  occur  until  after  an  exercise  or  event,  it  may  be 
more  difficult  to  elicit  and  capture  objective  data  in  the  future.  In  other  words,  the  further 
in  time  the  AAR  is  away  from  the  actual  event,  the  more  problematic  it  might  be  to 
remember  what  happened  accurately.  Nassim  Taleb  refers  to  this  problem  as  the  narrative 
fallacy.  This  fallacy 


5  Thomas  A.  Birkland,  “Disasters,  Lessons  Learned  and  Fantasy  Documents,”  Journal  of 
Contingencies  and  Crisis  Management  17,  no.  3  (2009):  146. 

6  Thomas  Mastaglio  et  al.,  “A  Study  of  Current  Practice  and  Theoretical  Foundations  for  the  After 
Action  Review,”  United  States  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  2010,  2. 

7  Ibid. 
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...addresses  our  limited  ability  to  look  at  sequences  of  facts  without 
weaving  an  explanation  into  them,  or,  equivalently,  forcing  a  logical  link, 
an  arrow  of  relationship,  upon  them.  Explanations  bind  facts  together. 

They  make  them  all  the  more  easily  remembered;  they  help  them  make 
more  sense.  Where  this  propensity  can  go  wrong  is  when  it  increases  our 
impression  of  understanding.8 

From  this  perspective,  the  AAR  becomes  a  narrative  of  what  has  happened  in  the  past 
based  on  sequences  of  remembered  facts.  Each  AAR  participant  engages  in  a  unique 
sense-making  process  to  piece  together  these  facts  so  that  the  entire  experience  has 
meaning.9  In  this  respect,  the  AAR  becomes  a  multitude  of  narratives  combined  as  one 
attempt  at  organizational  remembrance  and  understanding.  Thus,  whether  or  not  all  the 
explanations,  problems  and  solutions  crafted  to  make  sense  of  the  past  AAR  experience 
should  and  can  be  applied  to  a  future  experience  is  debatable. 

However,  the  fact  that  a  misnomer  has  literally  caused  confusion  regarding  the 
original  intent  of  the  process  may  be  the  biggest  problem  surrounding  the  AAR.  At  its 
inception,  the  AAR  was  designed  to  give  soldiers  instant  feedback  during  target 
practice. 10  In  its  original  design  and  use,  this  simple  practice  was  literally  an  after  action 
review  of  one  specific  action.  However,  AARs  are  now  used  to  help  various 
organizations  leam  from  a  number  of  much  more  complex  situations  that  contain  a 
variety  of  actions.  This  distinction  is  important.  An  exercise  or  emergency  is  composed 
of  multiple  actions  that  manifest  as  discrete  events,  such  as  the  completion  of  a  capability 
(or  associated  task),  a  decision  that  leads  to  a  certain  activity  or  the  communication  of  a 
message.  Thus,  a  more  correct  tenninology  for  the  process  as  it  is  now  conducted  would 
simply  be  the  After  Exercise  Review  (AXR)  or  After  Event  Review  (AER).  This  apparent 
contradiction  is  covered  more  thoroughly  in  a  later  chapter. 


8  Nassim  Nicholas  Taleb,  The  Black  Swan:  The  Impact  of  the  Highly  Improbable:  With  a  new  section: 
“On  Robustness  and  Fragility,  2nd.  ed.  (Random  House,  Inc.,  May  4,  2010),  Kindle  edition,  loc.  1818— 
1821. 

9  Karl  Weick,  Sensemaking  in  Organizations  (Thousand  Oaks,  CA:  Sage  Publications,  1995). 

John  E.  Morrison  and  Larry  L.  Meliza,  Foundations  of  the  After  Action  Review  Process,  (ARI 
Special  Report  42)  (Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences, 
ADA368651,  1999),  7. 
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Exercises,  or  real  events  for  that  matter,  are  rarely  formally  interrupted  (or 
stopped)  after  a  key  action  so  that  the  results  can  be  analyzed,  discussed,  and  acted  upon 
differently,  in  real  time,  to  test  for  a  different  outcome.  As  Baird,  Holland,  and  Deacon 
suggests,  “The  objective  [of  an  after  action  review]  is  to  learn  as  you  perfonn;  that  is  to 
understand  why  interim  objectives  were  not  accomplished,  what  lessons  could  be  learned, 
and  how  those  lessons  could  be  quickly  driven  back  into  the  perfonnance  process.”11 
Unfortunately,  this  process  is  not  usually  quick  if  achieved  at  all.  Ultimately,  the  larger 
problem  is  that  the  proper  individual  and  organizational  conditions,  and  support  for 
response  professionals  to  make  sense  of  and  act  upon  the  multiple  emergency  learning 
opportunities  that  occur  both  during  and  after  exercises,  and  real  emergencies,  have  not 
been  created. 

C.  BACKGROUND  AND  NEED 

The  capability  to  identify  and  exploit  these  opportunities  will  require  responders 
to  be  able  to  know  how  to  think  and  learn  both  during  and  after  the  emergency,  and  will 
also  require  them  to  manage  the  emergent  connections  between  the  two.  As  Moynihan 
points  out,  both  intercrisis  and  intracrisis  “learning  are  relevant  in  any  crisis  because 
response  leaders  will  seek  lessons  from  past  experience  but  also  make  ongoing 
assessments  about  the  effectiveness  of  the  current  response.”12  In  this  respect,  learning 
becomes  a  highly  adaptive  process  that  continually  occurs  both  during  and  after  an 
emergency. 

The  need  to  learn  both  during  and  after  a  crisis  suggests  that  the  traditional  way  of 
learning  reflected  by  the  current  use  of  the  AAR  must  be  reexamined.  The  complexity  of 
modem  day  exercises  and  emergencies  demands  that  responders  be  able  not  just  to  apply 
learning  from  past  events  but  also  to  reflect,  act,  and  leam  in  real  time.  Determining  how 
the  proper  individual  and  organizational  conditions  can  be  created  for  response 


1 1  Lloyd  Baird,  Phil  Holland,  and  Sandra  Deacon,  “Learning  from  Action:  Imbedding  More  Learning 
into  the  Performance  Fast  Enough  to  Make  a  Difference,”  Organizational  Dynamics  27,  no.  4  (Spring 
1999):  22. 

12  Donald  P.  Moynihan,  “Learning  under  Uncertainty:  Networks  in  Crisis  Management,”  Public 
Administration  Review’  68,  no.  2  (2008):  352. 
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professionals  to  make  sense  of  and  act  upon  the  various  “real  time”  crisis  learning 
opportunities  could  be  of  exponential  value  to  the  residents  of  this  country,  as  well  as  to 
all  the  men  and  women  who  serve  to  protect  them. 

D.  RESEARCH  QUESTION 

How  can  agencies  within  the  homeland  security  enterprise  create  the  individual 
and  organizational  conditions  to  help  promote  reflection,  action,  and  learning  both  during 
and  after  exercises  and  events  to  improve  the  overall  emergency  response? 

E.  METHOD 

This  thesis  briefly  traces  the  course  that  has  brought  adult  learning  from  its  early 
20th  beginnings  to  its  place  within  the  current  research  on  AARs  in  specific  and  crisis 
learning  theory  in  general.  Along  the  way,  it  focuses  on  relevant  theories  and  ideas,  such 
as  emergent  learning,  experiential  learning,  and  sense-making.  It  then  ties  these  theories 
and  concepts  together  into  an  emergency  learning  framework  that  can  be  used  to  help 
identify  and  make  sense  of  the  multiple  opportunities  for  reflection  and  action  both 
during  and  after  an  emergency. 
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II.  DEEP  BACKGROUND  AND  LITERATURE  REVIEW 


Chapter  II  is  divided  into  two  sections.  The  first  section  provides  a  deep 
background  on  the  AAR  and  its  related  approaches.  The  second  section  contains  a 
literature  review  that  focuses  on  the  AAR’s  theoretical  foundations. 

A.  DEEP  BACKGROUND  ON  THE  AAR 

1.  Definition  of  the  After  Action  Review 

In  its  most  basic  fonn,  an  AAR  is  an  evaluative  training  tool.  The  process  allows 
participants  to  receive  feedback  on  past  training  and  real-life  perfonnance  to  improve 
future  training  and  real-time  work-related  activities.  The  United  States  (U.S)  Army, 
credited  with  developing  the  AAR  in  the  mid-1970s,  describes  the  practice  in  its  Training 
Circular  (TC)  25-20,  A  Leader’s  Guide  to  After  Action  Reviews  as  “...a  professional 
discussion  of  an  event,  focused  on  perfonnance  standards,  that  enables  soldiers  to 
discover  for  themselves  what  happened,  why  it  happened  and  how  to  sustain  strengths 
and  improve  on  weaknesses.”13 

The  AAR  session  nonnally  occurs  immediately  after  an  exercise  or  real  event.  A 
facilitator  who  directs  the  experience  by  creating  an  open  conversational  forum  leads  the 
AAR.  The  facilitator  may  also  conduct  a  basic  question  and  answer  session  to  capture 
data  and  feedback  directly  from  the  participants.  A  more  thorough  review  of  the  process 
is  further  explained  later  in  the  chapter. 

2.  History  of  the  After  Action  Review 

The  foundations  of  the  process  have  been  traced  back  to  two  major  influences.14 
The  first,  S.L.A  Marshall’s  “interviews  after  combat,”  was  an  oral  history  approach  used 
during  World  War  II.  Marshall,  a  journalist  and  professional  historian,  would  gather 


13  Army  Combined  Arms  Center,  “A  Leader’s  Guide  to  After-Action  Reviews  (Training  Circular  25- 
20),”  1993,  http://www.au.af.mil/au/awc/awcgate/army/tc_25-20/tc25-20.pdf. 

14  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review  Process,  v. 
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soldiers  immediately  after  a  battle  and  pose  questions  designed  to  help  reconstruct  the 
course  of  events.  Marshall  and  his  colleagues  also  used  these  techniques  during  the 
Korean  and  Vietnam  wars. 15 

The  second  influence  was  the  “performance  critique.”  The  Army  used  this  review 
tool  to  provide  feedback  about  tactical  exercises  prior  to  the  1970s. 16  The  performance 
critique  relied  on  “human  umpires”  to  evaluate  the  outcomes  of  simulated  battles.  Due  to 
the  subjectivity  of  this  method,  the  feedback  had  the  potential  to  lack  real  credibility.  In 
addition,  the  critiques  were  often,  “...negative  in  tone,  thereby  fomenting  resentment 
among  participants  and  resistance  to  the  umpire’s  criticism.”17  Obviously,  this  type  of 
atmosphere  may  not  have  been  optimally  conducive  to  enhancing  individual  or  team 
perfonnance  and  learning. 

In  response  to  these  problems,  the  Anny  developed  the  Tactical  Engagement 
Simulation  (TES)  designed  to  provide  more  objective  feedback  to  trainees.  TES  was 
initially  configured  as  a  training  system  for  weapons  handling  in  which  lasers  were  used 
instead  of  live  ammunition.  This  type  of  laser-based  system  allowed  for  a  more  objective 
(and  safer)  way  to  measure  perfonnance  results  while  simulating  infantry  combat.  TES 
feedback  sessions  incorporated  Marshall’s  “interview”  technique  while  dampening  the 
negative  aspects  of  the  “performance  critique.”  This  new  feedback  methodology  became 
what  is  now  commonly  known  as  the  AAR. 18 

During  the  1980s  and  1990s,  the  Anny  developed  the  AAR  concept  as  a  way  to 
allow  for  quality  improvement  in  accord  with  organizational  restructuring  activities.19 
The  1993  “Leaders’  Guide  to  After  Action  Reviews”  continues  to  reinforce  the  practice 


15  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review  Process,  5-6. 

16  Ibid. 

17  Ibid.,  6. 

18  Ibid.,  7. 

19  Mastaglio  et  at,  “A  Study  of  Current  Practice  and  Theoretical  Foundations  for  the  After  Action 
Review,”  1. 


8 


by  stating,  “Competent  leaders  must  understand  and  apply  the  techniques  and  procedures 
which  produce  good  AARs.”20  Garvin  states  that  the  turning  point  for  the  AAR 

...was  the  Gulf  War.  AARs  sprang  up  spontaneously  as  small  groups  of 
soldiers  gathered  together,  in  foxholes  or  around  vehicles  in  the  middle  of 
the  desert,  to  review  their  most  recent  missions  and  identify  possible 
improvements.  Haiti  marked  a  further  step  forward.  There,  for  the  first 
time,  AARs  were  incorporated  into  all  phases  of  the  operation  and  were 
used  extensively  to  capture  and  disseminate  critical  organizational 
knowledge.21 

By  the  mid-1990s,  many  other  government  agencies  and  industries  had  adopted  the 
process  for  their  own  use. 

3.  Use  by  the  U.S,  Homeland  Security  Enterprise 

a.  Federal  Emergency  Management  Agency  (FEMA ) 

Any  state  emergency  management  agency  that  receives  an  Emergency 
Management  Perfonnance  Grant  (EMPG)  from  the  Federal  Emergency  Management 
Agency  (FEMA)  is  required  to  submit  an  AAR  after  an  exercise.22  Although  FEMA 
encourages  the  use  of  Homeland  Security  Exercise  and  Evaluation  Program  (discussed  in 
more  detail  later  in  the  chapter)  materials  to  direct  all  exercise  and  evaluation  activities, 
FEMA  does  not  require  any  standard  practice  on  how  to  conduct  the  AAR.  However, 
state  emergency  management  agencies  have  the  authority  to  require  local  emergency 
management  grantees  to  submit  AARs  in  a  standardized  fashion  if  so  inclined.23 

It  is  interesting  to  note  that  although  FEMA’s  own  program  policy 
stipulates  that  AARs  be  “scheduled  at  or  near  the  conclusion  of  emergency  or  disaster 
operations,”  The  Office  of  the  Inspector  General,  in  their  report,  “FEMA’s  Progress  in 


211  Army  Combined  Arms  Center,  “A  Leader’s  Guide  to  After-Action,”  ii. 

21  David  A.  Garvin,  “U.S.  Army’s  After  Action  Reviews:  Seizing  the  Chance  to  Learn,”  in  Learning  in 
Action,  A  Guide  to  Putting  the  Learning  Organization  to  Work  (Boston:  Harvard  Business  School  Press, 
2000),  106-116. 

22  FEMA,  “2011  Emergency  Management  Performance  Grant  Application  Kit,”  (n.d.), 
http://www.fema.gov/government/grant/empg/. 

23  Ibid. 
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Implementing  the  Remedial  Action  Management  Program,”  has  stated  that  the  agency 
itself  does  not  always  follow  the  requirement.24  Reasons  for  these  lapses  in  practice  vary; 
however,  they  include  budget,  timing,  and  personnel  issues.25 

Beyond  just  being  able  to  schedule  actual  AARs,  as  well  as  effectively 
capture  and  transfer  the  information  (as  discussed  earlier  in  this  report),  FEMA  has  also 
had  problems  with  developing  clear  and  concise  lessons  learned  and  best  practice 
guidance  that  may  arise  from  the  AAR  sessions.  The  Office  of  the  Inspector  General  cites 
an  example  in  which  “a  significant  amount  of  time  was  spent  discussing  the  lesson 
learned  and  best  practice  statements,  and  the  group  [finally]  detennined  that  two  of  the 
lessons  learned  where  in  fact  issues.”26 

b.  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP) 

The  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP) 
Volume  I  was  originally  released  in  2002  with  subsequent  volumes  (II-IV)  that  provide 
more  thorough  details  of  the  process.  A  major  rerelease  of  all  volumes  was  published  in 
2007.  The  purpose  of  HSEEP 

...is  to  provide  common  exercise  policy  and  program  guidance  that 
constitutes  a  national  standard  for  exercises.  HSEEP  includes  consistent 
tenninology  that  can  be  used  by  all  exercise  planners,  regardless  of  the 
nature  and  composition  of  their  sponsoring  agency  or  organization.  The 
volumes  also  provide  tools  to  help  exercise  managers  plan,  conduct,  and 
evaluate  exercises  to  improve  overall  preparedness.27 

The  HSEEP  approach  works  to  build  agency  capabilities  so  that  they  are  able  to  respond 
to  any  type  of  hazard  or  emergency.  Thirty-seven  capabilities  have  been  predefined  and 
are  identified  in  the  Target  Capabilities  List  (TCL),  and  the  Universal  Task  List  (UTL). 
According  to  the  HSEEP  Volume  III: 


24  Department  of  Homeland  Security,  Office  of  the  Inspector  General,  “FEMA’s  Progress  in 
Implementing  the  Remedial  Action  Management  Program,”  5. 

25  Ibid. 

26  Ibid.,  6. 

27  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP),  Volume  /,  February  2007,  v. 

10 


Capabilities  are  combinations  of  elements  (e.g.,  personnel,  planning, 
organization  and  leadership,  equipment  and  systems,  training,  exercises, 
assessments  and  corrective  actions)  that  provide  the  means  to  achieve  a 
measurable  outcome.28 

Exercises  are  evaluated  by  assessing  how  well  participants  meet  these  targeted 
capabilities  that  are  based  on  exercise  objectives.  Exercise  Evaluation  Guides  (EEGs)  are 
used  to  help  assess  the  objectives  by  linking  them  to  capability  tasks  and  activities. 
Exercise  evaluators  use  the  EEGs  to  “objectively  record  the  full,  partial,  or  non¬ 
completion  of  each  task.  The  EEG  is  a  reference  for  exercise  evaluators,  giving  a  sense  of 
when  activities  can  be  expected  to  occur  and  how  those  activities  relate  to  capability 
completion.”29 

Volume  III:  Exercise  Evaluation  and  Improvement  Planning  Materials 
outlines  an  eight-step  methodology  for  the  evaluation  of  exercises  and  the 
implementation  of  improvement  plans.  The  steps  include  the  following. 

1 .  Plan  and  Organize  the  Evaluation 

2.  Observe  the  Exercise  and  Collect  Data 

3.  Analyze  Data 

4.  Develop  the  Draft  After  Action  Report/Improvement  Plan 

(AAR/IP) 

5.  Conduct  After  Action  Conference 

6.  Identify  Corrective  Actions  to  be  implemented 

7.  Finalize  AAR/IP 

8.  Track  Implementation30 

It  is  interesting  to  note  that  the  phrase  “after  action  review”  does  not 
appear  in  any  of  the  HSEEP  volumes.  Instead,  the  materials  discuss  the  use  of  a  “player 
hot  wash.”  The  hot  wash 


28  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP),  Volume  III,  February  2007,  27. 

29  Ibid.,  4. 

30  Ibid. 
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...allows  players  to  engage  in  a  self-assessment  of  their  exercise  play  and 
provides  a  general  assessment  of  how  the  entity  performed  in  the  exercise. 

The  hot  wash  also  provides  evaluators  with  the  opportunity  to  clarify 
points  or  collect  any  missing  information  from  players  before  they  leave 
the  exercise  venue.31 

During  the  hot  wash,  players  are  also  asked  to  complete  Participant  Feedback  Forms  to 
“obtain  information  on  perceptions  of  the  exercise,  how  well  each  player  thought  his/her 
unit  performed,  and  how  well  the  unit  integrated  perfonnance  with  other  agencies  and 
other  exercise  components.”32  The  exercise  evaluation  team  then  uses  these  forms  to 
provide  insight  into  how  the  exercise  unfolded.  All  this  information  is  then  used  to  create 
an  After  Action  Report  that  the  Volume  III  materials  refer  to  as  an  AAR. 

As  can  be  imagined,  the  closely  related  tenninology  (including  two  AAR 
acronyms)  within  the  after  action  evaluation  discourse  can  be  confusing.  This  problem  is 
addressed  in  more  detail  later  in  this  report;  however,  for  now,  references  to  any  HSEEP 
related  activities  or  materials  will  be  referenced  in  complete  tenns  and  spelled  out  in  full 
(i.e.,  After  Action  Report,  hot  wash). 

In  this  researcher’s  review  of  the  general  AAR  literature,  the  tenn  “hot 
wash”  is  rarely  mentioned.  However,  in  step  5  of  the  HSEEP  methodology  (above),  the 
Volume  III  materials  dictate  the  use  of  an  After  Action  Conference  to  refine  the  After 
Action  Report  and  to  craft  an  improvement  plan  (IP).  According  to  Volume  III,  this 
conference  should  occur  within  one  month  of  exercise  completion.33 

As  previously  indicated,  HSEEP  guidelines  also  stipulate  the  use  of 
evaluators  during  the  exercise,  as  well  as  to  facilitate  the  hotwash.  Specifically,  the  role 
of  the  exercise  evaluator  is  to  observe  the  exercise  and  collect  data  on  elements  relevant 
to  the  EEGs,  such  as  specific  objectives,  completion  of  capabilities  and  tasks,  decision 
making,  etc.  This  information  is  then  compiled  into  the  draft  After  Action  Report. 


31  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP),  Volume  III,  14. 

32  Ibid. 

33  Ibid.,  19. 
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Although  Volume  III  contains  information  on  the  role  of  the  Exercise 
Facilitator,  no  direction  exists  on  how  to  facilitate  the  hotwash  or  the  After  Action 
Conference  effectively.  In  essence,  the  Volume  III  materials  state,  “A  hot  wash  is  led  by 
an  experienced  facilitator  who  can  ensure  that  the  discussion  remains  brief  and 
constructive,  and  who  can  focus  conversation  on  strengths  and  areas  for  improvement.”34 
The  lack  of  any  fonnalized  training  materials  seems  to  reinforce  some  of  the  concerns 
raised  by  Mastaglio  et  al.  referenced  earlier  in  this  report  regarding  a  lack  of  consistency 
in  facilitator  performance.35 

4.  The  Private  Sector 

The  idea  of  attempting  to  leam  formally  from  the  past  is  not  exclusive  to  the  U.S. 
Anny  or  the  U.S.  homeland  security  enterprise.  A  precursor  to  the  AAR  was  established 
at  Boeing  in  1969  when  it  instituted  Project  Homework.  This  initiative  was  an  attempt  to 
compare  the  problem-ridden  737  and  747  plane  programs  with  the  707  and  727  programs 
that  were  two  of  the  company’s  most  profitable  aircraft.  The  results  yielded  over  100 
recommendations  that  were  used  to  guide  the  757  and  767  development  programs, 
“...which  produced  the  most  successful,  error-free  launches  in  Boeing’s  history.”36 

According  to  Darling  and  Parry,  Shell  Oil  may  have  been  the  first  civilian  adopter 
of  the  traditional  Anny  AAR  method.37  The  authors  also  stress  that  Fidelity,  IBM,  and 
Harley  Davidson  have  operationalized  on-going  AAR  practices  within  different  units  of 
their  business  that  are  often  directed  by  ex-anny  officers.38  Today,  countless  private- 
sector  organizations  use  the  technique  in  one  form  or  another.  Even  Peter  Senge, 
consultant  for  many  high  profile  corporations  and  the  developer  of  the  “learning 
organization”  states,  “The  Anny’s  After  Action  Review  (AAR)  is  arguably  one  of  the 

34  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP),  Volume  III,  19. 

35  Mastaglio  et  al.,  “A  Study  of  Current  Practice  and  Theoretical  Foundations  for  the  After  Action 
Review.” 

36  David  A.  Garvin,  “Building  a  Learning  Organization,”  Harvard  Business  Review  (July-August 
1993):  85. 

37  Marilyn  Darling  and  Charles  Parry,  “After-Action  Reviews:  Linking  Reflection  and  Planning  in  a 
Learning  Practice,”  Reflections  3,  no.  2  (2001):  68. 

38  Ibid. 
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most  successful  organizational  learning  methods  yet  devised.”  However,  Senge  continues 
to  say,  “Yet,  most  every  corporate  effort  to  graft  this  truly  innovative  practice  into  their 
culture  has  failed  because,  again  and  again,  people  reduce  the  living  practice  of  the 
AAR’s  to  a  sterile  technique.”39 

5.  The  Standard  AAR  Method 

Although  not  indicated  for  use  by  the  HSEEP,  most  AAR  practices  are 
characterized  by  a  facilitative  technique  that  relies  upon  three  or  four  basic  open-ended 
questions.  The  1999  Special  Report  from  the  U.S.  Army  Research  Institute  (ARI)  listed 
these  questions  as  follows:  1)  What  happened  during  the  collective  training  experience? 
2)  Why  did  it  happen?  and  3)  How  can  units  improve  their  performance?40  These 
questions  can  be  modified  based  upon  the  type  of  organization  using  the  AAR.  For 
example,  a  2005  article  in  the  Harvard  Business  Review  suggests  that  corporations 
address  the  following  four  questions  in  the  AAR:  1)  What  were  our  intended  results?  2) 
What  were  our  actual  results?  3)  What  caused  our  results?  and  4)  What  will  we  sustain  or 
improve?41  The  researcher’s  own  experience  with  the  AAR  has  not  been  much  different. 
The  questions  that  his  agency  usually  asks  at  the  end  of  an  exercise  or  event  are  as 
follows:  1)  What  worked  well?  2)  What  were  the  gaps?  and  3)  What  can  we  do 
differently  next  time  to  improve  our  performance? 

Ultimately,  this  type  of  methodic  inquiry,  regardless  of  how  it  may  be  practiced, 
lends  itself  to  a  somewhat  linear  model  of  exploration  where  cause  and  effect  can  be 
assumed  within  the  questions  themselves.  The  questions  are  designed  for  an  ordered 
examination  of  the  exercise  or  event  that  may  hamper  an  understanding  of  its  true 
complexity.  Garvin  addresses  this  issue  by  stating: 


39  Peter  Senge,  “Introduction,”  in  From  Post-Mortem  to  Living  Practice:  An  In-Depth  Study  of  the 
Evolution  of  the  After  Action  Review,  ed.  Marilyn  Darling  and  Charles  Perry  (Boston,  MA:  Signet,  2001). 

40  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review  Process,  1 . 

41  Marilyn  Darling,  Charles  Parry,  and  Joseph  Moore,  “Learning  in  the  Thick  of  It,”  Harvard  Business 
Review  (July-August  2005):  4. 
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The  second  question  requires  that  participants  agree  on  what  actually 
happened  during  a  mission.  This  too  is  more  difficult  than  it  first  appears. 

Facts  can  be  slippery,  especially  when  stress  is  high  and  events  move 
rapidly.  All  too  often,  memories  are  flawed,  leading  to  competing  or 
inconsistent  stories.  Reality — what  soldiers  call  “ground  truth” — becomes 
difficult  to  pin  down,  resulting  in  gridlock  and  AARs  that  progress  slowly 
if  at  all.42 

What  Garvin  describes  is  a  variation  on  the  trap  of  Taleb’s  “narrative  fallacy.”  In  this 
respect,  the  AAR  becomes  a  type  of  story  that  an  organization  creates  around  an  exercise 
or  event  perfonnance  that  is  told  during  the  AAR.  Taleb  suggests  a  way  around  the 
narrative  fallacy,  “is  to  favor  experimentation  over  storytelling,  experience  over  history, 
and  clinical  knowledge  over  theories.”43  A  closer  look  at  these  ideas  in  relation  to  the 
AAR  and  overall  emergency  learning  occurs  later  in  the  thesis. 

6.  An  International  Perspective 

In  a  2009  paper  entitled  A  Review  of  After  Action  Review  Practice  for  Collective 
Training  in  the  British  Army,  produced  by  the  Human  Factors  Integration  Defence 
Technology  Centre,  (partly  funded  by  the  Human  Capability  Domain  of  the  U.K. 
Ministry  of  Defence  Scientific  Research  Programme)  the  authors  state: 

The  AAR  process  is  now  universally  acknowledged  as  an  effective 
process  and  “ provides  the  focus  for  virtually  all  collective  training ” 
(Morrison  and  Meliza,  1999,  p.  14).  As  such,  it  has  been  adopted  by  other 
services,  countries  and  for  use  outside  of  a  military  application  to  improve 
perfonnance  by  reflective  learning.44 

However,  like  many  North  American  writers  discussed  in  the  Literature  Review  section 
of  this  chapter,  it  seems  that  British  researchers  have  also  entered  the  AAR  echo 
chamber.  Validating  the  efficacy  of  the  AAR  process  through  a  pat  reference  to  the 
classic  Foundations  of  the  After  Action  Review  Process  by  Morrison  and  Meliza  occurs 
as  lazy  and  simple. 


42  Garvin,  “U.S.  Army’s  After  Action  Reviews:  Seizing  the  Chance  to  Learn,”  107-108. 

43  Taleb,  The  Black  Swan,  84. 

44  Human  Factors  Integration  Defence  Technology  Centre,  A  Review  of  After  Action  Review  Practice 
for  Collective  Training  in  the  British  Army,  June  2009,  Version  2. 
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Although,  concerning  the  above  statement,  the  authors  might  be  correct  in  one 
respect.  A  2003  paper  entitled  A  Comparative  Study  of  the  After  Action  Review  (AAR)  in 
the  Context  of  the  Southern  Africa  Crisis  does  seem  to  point  to  an  overall  adoption  of  the 
practice  worldwide.  This  paper  reviewed  three  case  studies  that  reflect  the  experiences  of 
World  Vision  International  (offices  worldwide),  the  British  Red  Cross  Society  (BRCS) 
and  the  consortium  of  Joint  Emergency  Food  Aid  Programme  (a  subsidiary  of  the  World 
Food  Programme  overseen  by  the  government  of  Malawi).45 

This  study  states  that  the  AAR  process  is  defined  by  the  following  four  questions: 
1)  What  was  the  objective  or  intent  of  the  action?  2)  What  went  well?  3)  What  went  less 
well  or  what  could  have  gone  better?  and  4)  What  would  we  do  differently  next  time?46 
These  questions  are  similar  to  the  process  outlined  in  the  2009  paper  describing  the 
British  model:  1)  What  was  supposed  to  happen?  2)  What  actually  happened?  3)  Why  did 
it  happen?  and  4)  What  can  be  learned  from  this  experience?47  As  indicated  by  the  earlier 
discussion  in  the  method  section,  this  somewhat  standard  questioning  technique  suggests 
that  the  AAR  process  is  practiced  in  a  fairly  similar  manner  across  the  world. 

However,  like  the  2009  paper  by  the  Human  Factors  Integration  Defence 
Technology  Centre,  the  2003  study  also  assumes  the  legitimacy  of  the  process  without 
delving  into  the  underlying  learning  literature  on  the  subject.  The  author’s  most  severe 
critique  of  the  practice  is  of  particular  interest: 

The  most  telling  observation  is  that  even  the  simplest  types  of  learning 
approach  can  be  difficult  to  orchestrate.  This  is  not  because  people  are 
incompetent  or  ill-informed,  it  is  because  work  environments  themselves 
are  extremely  complex  arrangements-the  task  of  getting  people  together, 
getting  ‘consent’  to  engage  in  collaborative  working  and  ensuring  that  the 
basic  logistics  of  an  event  are  catered  for,  often  leaves  little  room  for  the 
pre-event  preparation  and  post-event  follow-up.48 


45  Richard  Sexton  and  Isobel  McConnan,  A  Comparative  Study  of  the  After  Action  Review’  (AAR)  in 
the  Context  of  the  Southern  Africa  Crisis,  June  2003. 

46  Ibid. 

47  Human  Factors  Integration  Defence  Technology  Centre,  A  Review  of  After  Action  Review  Practice 
for  Collective  Training  in  the  British  Army. 

48  Sexton  and  McConnan,  A  Comparative  Study  of  the  After  Action  Review  (AAR)  in  the  Context  of  the 
Southern  Africa  Crisis ,  4 1 . 
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After  a  fairly  comprehensive  review  of  three  (not  so)  different  international  approaches  to 
the  ARR  practice,  the  authors  (surprisingly)  identify  logistics  as  the  AAR’s  most  serious 
problem.  Once  again,  an  assumption  of  efficacy  is  inherent  in  the  discussion  if  only 
“getting  people  together”  was  not  so  difficult.  As  the  following  literature  review  will 
reveal,  the  AAR  has  more  problems  than  the  pre-event  preparation  and  the  post-event 
follow  up. 

B.  THE  LITERATURE  REVIEW 

1.  A  Focus  on  the  Foundation 

As  discussed,  the  AAR  was  developed  by  the  U.S.  Anny  in  the  mid-1970s. 
Surprisingly,  for  a  process  developed  almost  40  years  ago,  not  much  is  known  about  how 
it  works  at  a  theoretical  level.  Most  of  the  literature  on  the  topic  relies  on  untested 
theoretical  claims  or  conjecture  while  scientific  research  is  thin  or  non-existent.  In  some 
ways,  the  practice  is  an  orphan  in  a  world  of  learning  without  a  foundational  home. 

This  review  of  the  literature  focuses  on  a  small  set  of  articles  and  reports.  The 
identified  literature  speaks  directly  to  the  AAR’s  theoretical  foundations.  Literature  that 
only  briefly  mentions  the  practice  or  attempts  to  discuss  it  without  tying  it  to  any  type  of 
learning  literature  was  not  considered  for  review. 

2.  What  We  Know 

The  U.S.  Anny  Research  Institute  (ARI)  for  the  Behavioral  and  Social  Sciences 
has  produced  most  of  what  is  known  about  the  theoretical  foundations  of  the  AAR.49  The 
ARI  will  often  contract  with  consulting  agencies  to  assist  in  authoring  their  reports.  For 
example,  MYMIC  LLC  contracted  to  produce  two  ARI  articles  mentioned  in  this  review 
and  The  Wexford  Group  contracted  to  do  one.  Since  researchers  on  this  topic  reside  in 
what  might  be  considered  a  somewhat  tight  research  community,  what  is  known  about 
the  topic  seems  to  exist  in  a  type  of  closed  loop  system  or  academic  echo  chamber.  The 
authors  have  a  tendency  to  reference  each  other  with  regards  to  work  done  on  the 

49  Margaret  S.  Salter  and  Gerald  E.  Klein,  After  Action  Reviews:  Current  Observations  and 
Recommendations  (Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences, 
2007). 
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theoretical  foundations  of  the  AAR  process.  However,  this  tendency  can  be  partly 
explained  because  the  literature  on  AAR  foundational  aspects  is  in  such  short  supply. 
Examples  of  this  phenomenon  are  more  closely  detailed  later  in  this  chapter. 

Also  known  is  that  the  AAR’s  theoretical  foundations  have  been  mostly  examined 
within  a  military  context.  Since  the  Army  developed  the  process,  it  seems  to  have  taken 
ownership  for  most  of  the  subsequent  research  done  on  its  underlying  theoretical 
foundations.  One  possible  reason  is  because,  “The  AAR  initially  evolved  in  reaction  to 
advancements  in  data  availability,  as  opposed  to  the  advancements  in  learning  science.”50 
As  described  earlier,  during  the  mid  1970s,  the  Anny  developed  TES  that  was  designed 
to  provide  more  objective  data  feedback  to  trainees.  TES  feedback  sessions  incorporated 
“interview”  techniques  derived  from  oral  history  approaches  used  during  World  War  II. 
This  new  feedback  methodology  became  what  is  now  commonly  known  as  the  AAR.51 

Thus,  it  was  only  after  the  development  of  the  actual  AAR  that  the  literature 
(driven  by  the  ARI)  truly  begins  to  explore  the  educational  theory  supporting  its  use.  For 
example,  the  ARI  published  the  Foundations  of  the  After  Action  Review  Process  in  July 
1999.  The  report,  written  by  John  Morrison  and  Larry  Meliza,  is  considered  by  many  to 
be  the  most  referenced  resource  regarding  the  practice.52  The  document  provides  a 
thorough  historical  examination  and  of  particular  interest  to  this  review  states  that  the 
purpose  of  its  Conceptual  Foundations  section  is  to  “...identify  and  document  the 
behavioral  science  theories  and  principles...”  that  underlie  the  AAR  practice.53  This 
attempt  represents  the  first  serious  effort  to  determine  the  theoretical  foundations  of  the 
AAR  practice.  However,  in  a  nod  to  the  difficulty  of  their  endeavor,  the  authors  quickly 
follow  their  stated  purpose  with  a  type  of  disclaimer  that  suggests  the  actual  intention  was 


50  Thomas  W.  Mastaglio,  Philip  Jones,  Elizabeth  T.  Newlin,  and  James  P.  Bliss,  An  Integrated  Theory 
for  After  Action  Review,  Submitted  to  the  tnterservice/lndustry  Training,  Simulation,  and  Educational 
Conference  (EITSEC),  Paper  #7361,  2007,  3. 

51  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review  Process,  5-6. 

52  James  P.  Bliss,  Steven  A.  Minnis,  Jeffery  Wilkinson,  Thomas  Mastaglio,  and  John  S.  Barnett, 
“Establishing  an  Intellectual  and  Theoretical  Foundation  for  the  After  Action  Review  Process-A  Literature 
Review,”  United  States  Army  Research  Institute,  April  2011,  24. 

53  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review  Process,  23. 

18 


not  to  deliver  a  comprehensive  tally,  but  to  “...provide  an  appreciation  for  the  range  and 
variety  of  the  theoretical  concepts  that  have  been  incorporated  into  the  AAR.”54 

Regardless,  the  authors  identified  the  following  six  general  areas  of  theory  and 
research  as  being  associated  with  the  practice:  information  feedback,  performance 
measurement,  memory  and  cognition,  group  processes  and  dynamics,  communication 
theory  and  techniques,  and  instructional  science. 

However,  although  Morrison  and  Meliza’s  work  could  be  considered  an  initial 
attempt  to  identify  the  foundational  theory  of  the  AAR,  subsequent  literature  seems  to 
enshrine  their  claims  as  established  fact.  For  example,  ARI  produced  a  report  in  January 
2007  entitled  After  Action  Reviews:  Current  Observations  and  Recommendations.  The 
document,  authored  by  Salter  and  Klein  of  The  Wexford  Group  states,  “Morrison  and 
Meliza  (1999)  provided  part  of  the  reason  for  the  AAR’s  success  by  showing  its  basis  in 
behavioral  science,  citing  feedback,  perfonnance  measurement,  memory,  group 
dynamics,  communication,  and  instruction.”55 

This  presumption  of  “success”  could  have  the  undesired  effect  of  cutting  off 
future  attempts  of  researching  the  theoretical  foundations  question  in  any  further  detail. 
Just  because  it  showed  its  basis  does  not  mean  any  science  was  applied  to  prove  its  basis. 
Indeed,  a  September  2010  report  produced  by  MYMIC  LLC  and  written  by  Mastaglio  et 
al.  entitled  A  Study  of  Current  Practice  and  the  Theoretical  Foundations  for  the  After 
Action  Review  adds  weight  to  the  possibility  that  this  initial  effort  might  mistakenly  be 
accepted  as  gospel  within  the  field.  The  report  states,  “Most  of  the  military  AAR  research 
in  the  past  10  years  has  shifted  away  from  foundational  and  theoretical  work  to 
application  of  the  techniques  under  emerging  training  technology  conditions  or 
development  of  technologies  to  support  the  established  AAR  process.  The  fundamental 
research  conducted  prior  to  1999  has  been  well  documented  and  described  in  other  works 
(Salter  &  Klein,  2007;  Mastaglio,  Jones,  Bliss  &  Newlin,  2007). ”56  In  actuality,  Salter 

54  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review’  Process,  23. 

55  Salter  and  Klein,  After  Action  Reviews:  Current  Observations  and  Recommendations,  5. 

5®  Mastaglio  et  al.,  “A  Study  of  Current  Practice  and  Theoretical  Foundations  for  the  After  Action 
Review,”  7. 
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and  Klein  only  cite  the  previously  discussed  work  of  Morrison  and  Meliza  concerning  a 
theoretical  foundation.57  Thus,  these  authors  seem  to  be  perpetuating  a  rehash  of  the 
same  infonnation.  In  this  sense,  the  claims  exist  within  a  type  of  academic  echo  chamber 
around  the  theoretical  foundations  inquiry  and  appear  to  be  potentially  shutting  down  the 
question. 

However,  another  2007  publication  authored  by  Mastaglio  and  Jones  of  MYMIC 
and  Bliss  and  Newton  of  Old  Dominion  University,  does  attempt  to  address  the  issue 
again.  The  paper,  An  Integrated  Theory  for  After  Action  Review,  was  submitted  to  the 
2007  Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC). 
The  authors,  as  with  the  other  attempts,  explored  the  question  from  a  military  perspective 
and  claimed  that  the  “...AAR  needs  a  practical  theoretical  foundation  that  can  be  used  as 
a  basis  for  teaching  it,  infonning  the  development  of  AAR  tools,  and  guiding  future 
cognitive  science  learning  research  as  it  relates  to  training  approaches.”58  Their  proposed 
solution  is  to  present  an  integrated  theoretical  model  based  on  learning  theory,  social 
psychology  and  cognitive  science  that  will  serve  as  a  foundation  for  further  research.  The 
theory  attempts  to  integrate  “relevant  behavioral  constructs,  including  goals,  feedback, 
knowledge,  and  performance  appraisal.”59  Once  the  theory  is  introduced,  the  authors 
explain  how  each  of  these  themes  applies  to  one  another  by  referencing  research  “about 
how  humans  learn,  coordinate,  integrate  feedback  and  perform  effectively.”60 

This  approach,  where  behavioral  tenets  of  the  AAR  are  identified  (i.e.,  goals, 
feedback,  knowledge,  etc.)  and  then  supported  by  research  that  shows  their  individual 
applicability  in  other  domains,  acts  as  somewhat  of  a  stand  in  for  original  foundational 
research  on  the  actual  AAR  itself.  Ultimately,  as  suggested  earlier,  these  types  of  after- 
the-fact  efforts  to  describe  the  underlying  theory  seem  like  a  stab  at  lifting  a  house  off  the 
ground  to  apply  a  foundation  that  never  originally  existed.  To  complete  the  metaphor — if 
the  AAR  were  indeed  a  house,  it  was  constructed  on  shaky  ground. 

57  Salter  and  Klein,  After  Action  Reviews:  Current  Observations  and  Recommendations,  5. 

58  Mastaglio  et  at,  An  Integrated  Theory  for  After  Action  Review,  2. 

59  Ibid.,  6. 

60  Ibid.,  9. 
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To  be  clear,  explaining  natural  phenomenon  when  appropriate  evidence  exists  to 
demonstrate  its  validity  and  reliability  is  what  science  is  all  about.  However,  this  has  not 
been  the  case  with  the  AAR.  The  theories  mentioned  above  are  just  that.  In  essence,  they 
are  a  way  to  make  sense  of  how  the  AAR  process  works.  The  problem  is  that  beyond 
anecdotal  evidence,  it  is  still  not  known  for  sure  how  it  works  or  if  it  even  does  work 
because  the  theories  have  not  been  substantially  tested.  Ultimately,  as  Phillip  Jones  (co¬ 
author  of  the  2010  MYMIC  report  referenced  earlier),  in  an  email  sent  to  the  researcher 
on  July  5,  2011,  regarding  whether  or  not  the  AAR  achieves  its  desired  result,  concedes 
“that  there  has  not  been  much  research  into  the  efficacy  of  the  process.”61 

3.  What  is  Unknown? 

Therefore,  what  is  known  basically  on  this  subject  is  that  currently  prescribed 
theoretical  underpinnings  have  not  been  adequately  expressed,  explored,  or  assessed.  One 
of  the  most  recent  publications  on  the  subject  again  supports  this  observation.  An  April 
2011  ARI  produced  document  entitled  Establishing  an  Intellectual  and  Theoretical 
Foundation  for  the  After  Action  Review  Process — A  Literature  Review  captures  the 
problem  well  by  stating,  “To  date,  few  theoretical  models  have  been  proposed.  Those  that 
have  been  proposed  suffer  from  a  lack  of  validation.  Thus  the  irony:  even  though  AARs 
are  meant  to  be  the  pinnacle  of  a  ‘learning  organization,’  there  have  been  few 
documented  attempts  to  specify  the  principles  of  learning  or  knowledge  management  to 
conduct  the  ARR  sessions.”62 

Although  this  researcher  takes  exception  to  the  reference  about  the  AAR  being  the 
“pinnacle  of  a  learning  organization,”  the  overall  intent  of  the  statement  occurs  as 
essentially  valid. 

Another  related,  yet  unexplored  area  within  the  literature  is  whether  or  not  the 
AAR  practice  is  based  upon  pedagogy  or  andragogy.  Pedagogy  concerns  itself  with  a 
general  application  to  the  study  of  teaching  and  learning  and  has  been  primarily 

61  Phillip  Jones,  e-mail  message  to  author,  July  5,  2011,  Portsmouth,  VA. 

62  Bliss  et  ah,  “Establishing  an  Intellectual  and  Theoretical  Foundation  for  the  After  Action  Review 
Process-A  Literature  Review,”  vi. 
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associated  with  those  activities  in  relationship  to  children.  The  process  is  instructor  led 
and  puts  the  student  in  a  submissive  position.  Andragogy  is  concerned  specifically  with 
how  adults  learn  and  is  characterized  by  a  self-directed  and  experience-centered 
orientation  to  learning.63  Due  to  the  relative  youth  of  those  recruited  into  the  military, 
pedagogy  is  a  feasible  teaching  and  learning  orientation  within  that  environment. 

However,  an  andragogical  orientation  is  also  potentially  valid.  A  recent  article  in 
the  June  14,  2010,  Tampa  Bay  Times  states,  “In  2005,  71.2  percent  of  Army  recruits  were 
21  or  younger.  In  2009,  it  was  down  to  55.9  percent.”64  In  other  words,  in  2009,  44.1% 
of  Anny  recruits  were  older  than  2 1 .  Regardless,  from  a  homeland  security  perspective, 
the  take  away  is  that  the  enterprise  is  comprised  of  learners  of  all  ages  and  must  at  least 
be  considered  from  an  adult  learning  perspective. 

However,  much  of  the  AAR  literature  refers  to  the  process  from  a  pedagogical 
perspective.  For  example,  Morrison  and  Meliza  state,  “Because  the  AAR  is  a  pedagogical 
process,  the  principles  of  instructional  science  should  be  used  to  enhance  the 
effectiveness  of  the  AAR.”65  Mastaglio  et  al.  concur  with  this  view  and  state,  “AARs  are 
meant  to  serve  a  pedagogical  purpose:  to  enhance  learning  from  experience.”66  Finally, 
the  2011  ARI  document  referred  to  earlier  in  this  section  categorically  states,  “The 
philosophy  of  AARs  is  pedagogical.”67 

In  an  email  to  Jeffery  Wilkinson,  one  of  the  authors  of  the  2011  ARI  document, 
the  researcher  asked  if  he  and  the  other  authors  were  intentional  in  their  use  of  the  term 
pedagogy.  His  response,  received  on  July  6,  201 1,  is  as  follows: 


63  Adapted  and  Updated  from  Moving  from  Pedagogy  to  Andragogy,  Roger  Hiemstra  and  Burton 
Sisco,  Individualizing  Instruction  (San  Francisco:  Jossey-Bass,  1990),  http://www- 

distance .  syr.  edu/andraggy .  html . 

64  Hilary  Lehman,  “Army  Welcomes  Older  Recruits  Looking  for  Stability,”  Tampa  Bay  Times,  June 
14,2010. 

65  Morrison  and  Meliza,  Foundations  of  the  After  Action  Review  Process,  5 1 . 

66  Mastaglio  et  al.,  “A  Study  of  Current  Practice  and  Theoretical  Foundations  for  the  After  Action 
Review,”  2. 

67  Bliss  et  al.,  “Establishing  an  Intellectual  and  Theoretical  Foundation  for  the  After  Action  Review 
Process-A  Literature  Review,”  2. 
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As  for  pedagogy  vs.  andragogy-our  use  of  pedagogy  was  not  purposeful 
but  instead  out  of  a  lack  of  grounded  awareness  of  the  distinction.  In  the 
training  world  we  work,  we  are  trainers,  technologists,  and  behavioral 
psychologists,  the  term  pedagogy  has  been  (is)  the  (inappropriate?)  tenn 
of  art.  Now  that  I’ve  quickly  explored  the  definitions  and  read  a  discussion 
of  the  distinctions  between  them  I  see  where  andragogy  may  be  the  more 
correct  tenn.  However,  I  do  intend  to  explore  this  a  bit  further  as  what  I 
read  leads  me  to  believe  that  there  are  details  in  the  definitions  that  may 
not  just  lie  along  the  lines  of  child  vs  adult  education.  [Sic]68 

Wilkinson  in  essence  states  he  is  unaware  of  the  distinction.  The  use  of  the  tenn  is 
not  intentional  but  comes  from  a  lack  of  knowledge  about  the  concept.  In  this  respect,  the 
introduction  of  this  theory  to  a  primary  author  in  the  field  is  occuning  as  something  that 
is  unknown.  However,  it  is  interesting  to  note  that  although  Wilkinson  describes  him  and 
his  colleagues  as  “trainers,  technologists,  and  behavioral  psychologists”  he  does  not  use 
the  term  researcher. 

This  omission  may  point  to  one  of  the  larger  issues  related  to  the  AAR  theoretical 
foundations  question.  This  review  has  revealed  that  consultants  and  practitioners  who 
have  an  interest  in  the  theoretical  aspects  of  the  process  are  producing  much  of  the 
literature.  This  researcher  proposes  that  the  practical  orientation  of  other  researchers  in 
the  field  may  be  limiting  their  abilities  in  doing  the  thorough  investigation  that  the  AAR 
problem  warrants.  Thus,  the  pedagogy/andragogy  question  may  suffer  from  the  same 
closed-loop  phenomenon  described  earlier.  The  combination  of  researchers  with  a 
possible  lack  of  research  skills  plus  a  community  that  may  be  operating  in  a  closed  loop 
echo  chamber  is  potentially  limiting  the  ability  of  the  field  to  tap  in  to  new  and 
unexplored  areas  of  research  and  inquiry. 

a.  What  Should  We  Know? 

Moynihan’s  article  From  Intercrisis  to  Intracrisis  Learning  proposes  a 
new  way  of  thinking  about  learning  from  emergencies.  As  suggested  earlier,  Moynihan 
would  describe  the  AAR  practice  as  a  type  of  intercrisis  learning  that  is  a  way  of 


68  Jeffrey  Wilkinson,  e-mail  message  to  author,  July  6,  2011,  Oviedo,  FL. 


23 


“learning  from  one  crisis  to  prepare  for  another.”69  However,  he  distinguishes  this 
concept  from  intracrisis  learning  “...that  seeks  to  improve  response  during  a  single  crisis 
episode.”70 

Deverell  continues  the  investigation  into  the  idea  of  intracrisis  learning  in 
his  article  Crises  as  Learning  Triggers:  Exploring  a  Conceptual  Framework  of  Crisis- 
Induced  Learning.  Deverell  suggests,  “Crisis-induced  learning  triggers  tell  organizational 
members  that  they  need  new  routines  and  procedures  to  manage  the  events  at  hand.  These 
acute  crisis  characteristics  make  intracrisis  learning  more  difficult  than  intracrisis 
learning  (Dror,  1988;  Lagadec,  1990),  which  happens  when  there,  is  in  general,  more 
time  to  contemplate.”71 

What  is  interesting  about  these  articles  is  that  both  Deverell  and  Moynihan 
attempt  to  connect  their  ideas  about  both  inter  and  intra  crisis  learning  within  the 
contemporary  organizational  learning  literature.  Specifically,  they  reference  the  work  of 
Argyris  and  Schon  on  distinguishing  between  single  and  double-loop  learning  in  both 
papers.72  Naot,  Lipshitz,  and  Popper  state  that  double  loop  learning  is,  “considered  to  be 
of  higher  quality  because  effective  solution  of  some  problems  requires  the  examination  of 
sensitive  undiscussable  issues,  and  the  refraining  of  assumptions,  values  and  goals.”73 
Whereas  single  loop  learning  is  more  interested  in  a  quick  fix,  double  loop  considers  the 
larger  context  and  works  to  shift  organizational  culture  (values,  beliefs,  assumptions,  etc.) 
when  necessary  to  truly  implement  a  lesson,  and  more  importantly,  change  individual  and 
organizational  behavior.  For  example,  Moynihan  states,  “The  creation  of  the  ICS  can  be 


69  Moynihan,  “From  Intercrisis  to  Intracrisis  Learning,”  189. 
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71  Edward  Deverell,  “Crises  as  learning  Triggers:  Exploring  a  Conceptual  Framework  of  Crisis- 
Induced  Learning,”  Journal  of  Contingencies  and  Crisis  Management  17,  no.  3  (September  2009):  182. 
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considered  an  example  of  intercrisis  double-loop  learning,  as  it  shows  practitioners  and 
policy  makers  questioning  basic  approaches  to  crisis  response,  and  developing  a  new 
framework  for  future  responses.”74 

b.  Conclusion 

Ultimately,  this  somewhat  critical  analysis  of  the  AAR  learning  literature 
does  not  effectively  dismiss  the  efficacy  of  the  AAR  practice  or  of  the  subsequent 
attempts  to  identify  a  foundational  home.  To  be  clear,  many  anecdotal  instances  within 
the  general  AAR  literature  attest  to  its  success  and  viability.  As  discussed  later  in  this 
thesis,  the  AAR  does  have  a  place  (albeit  in  a  revised  form)  within  an  emergency  learning 
framework.  However,  the  concerns  addressed  in  this  review  of  the  learning  literature 
along  with  those  voiced  by  researchers,  such  as  Donahue  and  Tuohy,  suggest  that  the 
AAR,  at  least  within  the  homeland  security  enterprise,  needs  to  be  more  closely 
examined. 

In  other  words,  the  problems  identified  in  this  review  of  the  literature 
imply  that  traditional  ways  of  thinking  about  how  individuals  and  organizations  leam 
from  an  exercise  or  an  actual  emergency  must  be  explored  further.  Conceptualizing  the 
AAR  from  the  perspective  of  different  theories  and  methodologies  that  allow  for  non¬ 
linear  approaches  may  help  to  move  the  practice  forward.  A  brief  review  of  some  these 
practices,  as  couched  within  the  knowledge  management  (KM)  literature,  is  discussed  in 
the  next  chapter. 


74  Moynihan,  “From  Intercrisis  to  Intracrisis  Learning,”  196. 
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III.  STORIES  AND  SENSEMAKING 


As  discussed  in  previous  chapters,  the  type  of  methodic  inquiry  that  the  traditional 
AAR  espouses,  regardless  of  how  it  may  be  practiced,  lends  itself  to  a  somewhat  linear 
model  of  exploration  where  cause  and  effect  can  be  assumed  within  the  questions 
themselves  (what  happened,  what  was  suppose  to  happen,  etc.).  In  other  words,  the 
questions  are  designed  for  an  ordered  examination  of  the  exercise  or  event.  Although  this 
type  of  inquiry  may  have  value  generating  feedback  specific  to  identified  objectives,  the 
technique  is  limited  in  dealing  with  the  more  complex  issues  related  to  organizational 
learning  and  knowledge  generation.  For  the  purposes  of  this  thesis,  organizational 
learning  is  defined  “as  a  process  of  improving  organizational  actions  through  better 
knowledge  and  understanding... a  useful  conception  of  organizational  learning  must 
include  change,  such  that  an  organization  can  be  said  to  leam  when  its  actions  have  been 
modified  as  a  result  of  reflection  on  new  knowledge  or  insight.”75  In  other  words,  new 
knowledge  leads  to  changes  in  organizational  behavior. 

A.  A  NEW  MODEL  AAR 

Some  researchers  and  practitioners  are  looking  beyond  the  traditional  behavioral 
science  theories  and  techniques  that  have  dominated  the  AAR  conversation  since  its 
inception.  These  individuals  are  using  alternative  facilitative  theories  and  tools  that  exist 
outside  of  traditional  behavioral  science  models  and  practices.  Specifically,  the  use  of 
narrative  (or  story),  as  it  exists  within  the  field  of  (KM),  is  an  example  of  one  such 
approach  and  is  described  later  in  the  chapter. 

1.  A  Knowledge  Based  Solution 

Before  the  researcher  begins  a  discussion  on  alternative  techniques,  it  is  important 
to  define  the  concept  of  KM.  Oliver  and  Snowden  describe  the  discipline  in  the  following 
way: 


75  Amy  C.  Edmondson,  “The  Local  and  Variegated  Nature  of  Learning  in  Organizations:  A  Group 
Level  Perspective,”  Organization  Science  13,  no.  2  (2002):  128. 
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In  general,  the  focus  has  been  on  the  process  of  abstraction  of  knowledge 
from  the  heads  of  knowledge  owners  into  some  codified  form  either  for 
use  in  a  process,  or  to  enable  the  ‘knowledge’  to  be  internalized  by  other 
humans.  Coupled  with  the  growth  of  distributed  computing,  collaboration 
software  and  advertising  slogans  such  as  ‘information  at  your  finger  tips’, 
this  has  led  to  an  increasing  focus  on  abstract  knowledge,  capable  of  rapid 
diffusion  within  a  system  independently  of  the  human  who  formerly 
possessed  the  knowledge.76 

It  is  interesting  to  note  that  in  August  2008,  the  Anny  released  FM  6-01.1, 
Knowledge  Management  Section  that,  “establishes  the  doctrinal  principles,  tactics, 
techniques,  and  procedures  necessary  to  effectively  integrate  KM  into  the  operations  of 
brigades,  divisions,  and  corps.”77  This  field  manual  represents  the  first  time  that  the 
Anny  addresses  the  discipline  of  KM.  Within  this  document,  the  AAR  is  referenced  and 
described  as  a  process,  “designed  to  provide  feedback  on  performance  during  exercises 
by  involving  participants  in  the  training  diagnostic  process.  Involving  participants 
increases  and  reinforces  learning.”78  Storytelling  is  listed  as  a  learning  technique  that: 

...helps  communicate  complicated  ideas,  situations,  and  experiences.  It 
helps  Soldiers  and  units  understand  and  recreate  a  mental  framework  for 
learning.  Storytelling  enables  an  organization  to  see  itself  differently, 
make  decisions,  and  change  behaviors  in  accordance  with  these  new 
perceptions,  insights  and  identities.79 

Unfortunately,  the  document  fails  to  go  into  any  further  detail  on  how  to  facilitate 
“storytelling”  techniques  or  practices. 

Fortunately,  narrative  techniques  have  been  identified  and  elucidated  within  the 
larger  KM  literature  and  are  available  to  facilitators  when  an  organization  is  interested  in 
focusing  on  knowledge  discovery  beyond  the  usual  routine  of  determining  whether  or  not 
an  exercise  or  event  objective  has  been  met.  David  Snowden,  in  a  three  part  series  of 
articles  originally  published  in  1998  and  reissued  in  2005,  lays  out  an  intriguing  set  of 
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such  tools.  The  techniques  are  “derived  from  anthropology  and  based  on  the  organizing 
principle  that  ‘We  only  know  what  we  know  when  we  need  to  know  it.’”80 

Before  a  brief  review  of  what  the  researcher  considers  some  of  Snowden’s  more 
applicable  AAR-like  knowledge  generating  tools,  an  attempt  to  define  “knowledge” 
further  would  seem  advisable.  However,  because  his  intention  in  this  thesis  is  not  to  delve 
into  a  philosophical  discussion  of  epistemology,  he  reminds  the  reader  of  Snowden’s 
earlier  “organizing  principle”  and  adds  that  knowledge  “it  is  triggered  by  events  and  by 
need.”81  Snowden  also  suggests,  “Knowledge  is  simultaneously  and  paradoxically  a  thing 
and  a  capability,  an  actuality  and  a  potential,  tangible  and  intangible.”82  With  that  said, 
Snowden  states,  “Knowledge  Disclosure  Points  (KDPs)  comprise  decisions,  judgments, 
problem  resolution  and  learning.  They  are  the  points  at  which  we  use  knowledge.”83 
Thus,  if  people  are  left  unsure  of  what  knowledge  actually  is,  at  least  they  know  when 
they  use  it. 

However,  to  go  beyond  Snowden’s  more  philosophical  musings  is  to  break 
knowledge  down  into  either  tacit  or  explicit  knowledge.  Explicit  knowledge,  or  “know- 
what”  is  knowledge  that  is  codifable,  which  is  to  say  that  it  can  be  documented  in  plans, 
procedures,  instruction  manuals,  etc.  Tacit  knowledge  or  “know-how”  is  available,  as 
Snowden  suggests  when  it  needs  to  be  known.  This  type  of  knowledge  exists  as  more  of  a 
gut  instinct  that  is  spontaneous  and  generated  from  years  of  experience  and  cannot  be 
documented  or  captured  in  a  way  easily  accessed  or  understood  in  written  form.84 


80  David  Snowden,  “The  ASHEN  Model:  An  Enabler  of  Action,”  Knowledge  Management  3,  no.  7 
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Snowden  discusses  the  use  of  KDPs  within  the  context  of  the  ASHEN  model  that 
“was  created  as  a  means  of  providing  a  linguistic  framework  both  to  help  organizations 
identify  what  they  know  and  to  move  directly  to  action  as  a  result  of  the  meaning 
provided  by  the  language.”85 

The  five  ASHEN  components  are  artefacts,  skills,  heuristics  experience  and 
natural  talent.  These  component  pieces  represent  the  “knowledge  assets”  of  an 
organization.  Artefacts  (the  naturally  created  “things”  of  an  organization),  skills 
(expertise)  and  heuristics  (rules  of  thumb)  are  examples  of  explicit  knowledge  (able  to  be 
documented)  and  somewhat  manageable.  Experience  and  natural  talent  are  more  tacit  in 
nature  and  not  as  easy  to  codify  and  control.86 

Within  an  After  Action  Review  setting,  a  discussion  of  the  components  in 
relationship  to  KDPs  could  help  to  achieve  meaningful  responses.  For  example,  during  an 
AAR,  an  incident  commander  might  be  asked  about  when  she  solved  a  certain  problem 
during  an  exercise  or  event.  The  answer  could  include  a  discussion  of  artifacts  or 
heuristics  used  to  produce  a  solution.  Other  questions  might  focus  on  judgments  made  in 
the  field  by  first  responders  that  required  specific  previous  experiences,  natural  talents  or 
skills.  These  types  of  questions,  as  opposed  to  the  standard  set  of  three  or  four  described 
earlier  in  this  thesis,  have  the  ability  to  tap  into  unique  knowledge  sets  that  can  be 
documented  so  that  an  organization  is  able  to  move  “from  key-person  dependency  to 
knowledge  dependency.  This  essential  step  of  depersonalization  is  critical  to  effective 
knowledge  practice.”87 

The  value  of  “knowledge  dependency”  becomes  acutely  evident  when  a  long-term 
employee  leaves  an  organization.  Many  times  this  person  will  take  the  “organizational 
knowledge”  with  her  and  leave  a  huge  void.  A  corresponding  problem  within  the  field  of 
emergency  management  is  when  a  new  incident  management  team  cycles  through  an 
exercise  or  emergency  and  the  previous  team  cycles  out.  Although  various  incident 
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command  forms  are  used  to  document  past  work  during  the  shift,  “knowledge  assets”  that 
are  gathered  during  a  shift  debrief  could  be  used  to  get  new  team  members  quickly  up  to 
snuff. 

Another  narrative  knowledge  gathering  tool  that  Snowden  proposes  is 
anthropological  observation  in  which  observers  seek  to  blend  in  to  their  surroundings 
“before  they  form  a  hypothesis.”88  Traditional  exercises  and  AARs  also  use  observer  and 
evaluator  feedback.  However,  the  difference  in  Snowden’s  model  is  that  his  observers  are 
not  initially  looking  for  anything  in  particular.  Unlike  traditional  exercise  observers  who, 
with  clip  board  in  hand,  are  looking  to  identify  specific  objectives,  tasks  (capabilities)  or 
behaviors  related  to  the  exercise  objectives,  Snowden’s  observers  are  encouraged  to, 
“Stack  vegetables,  carry  bags,  make  tea,  sweep  metal  scarf  off  the  factory  floor  with 
apprentices,  dig  holes  in  the  road...”89  The  idea  is  that  the  more  the  observer  can  become 
a  camouflaged  part  of  the  scene,  the  better  able  to  possibly  observe  and  note  KDPs. 
Snowden  suggests  that  if  the  observed  KDP  is  “clear  and  distinct  then  the  ASHEN 
question  may  be  directly  asked:  ‘What  Artifacts,  Skills,  Heuristics,  Experience  and 
Natural  Talent  were  necessary  when  you  made  that  decision?”’90 

Snowden  also  proposes  the  specific  use  of  story  as  a  disclosure  tool.  In  this 
technique,  for  example,  incident  management  teams  would  be  encouraged  to  create  both 
the  “official  history”  of  the  response,  as  well  as  alternative  versions.  The  alternative 
versions  are  used  to  “disrupt”  the  official  history  to  help  reveal  hidden  truths.  For 
example,  Snowden  proposes  the  use  of  a  technique  called  alternative  histories.91  In  this 
technique,  an  incident  management  team  would  initially  create  an  official  history  of  its 
response  or  recovery  efforts.  After  this,  “...they  are  then  sent  back  to  identify  turning 
points  in  that  history  where  a  minor  change  may  have  resulted  in  failure.”92  The  teams 
would  then  be  asked  to  create  a  new  story  based  on  each  turning  point.  With  a  number  of 
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alternative  histories,  KDPs  and  ASHEN  components  can  be  used  to  mine  for  knowledge 
within  a  variety  of  contexts.  The  stories  allow  the  participants  to  enter  into  a  type  of 
alternative  reality  where  knowledge  is  triggered  regardless  of  whether  parts  of  the  event 
actually  happened  or  not. 

B.  SENSEMAKING 

Stories  are  considered  by  many  to  be  a  way  to  make  sense  of  what  happens  in 
organizations.  For  example,  Weick  suggests  that  the  role  of  narrative  as  it  relates  to 
sensemaking  is  pivotal  to  organizational  learning.93  Snowden,  who  is  also  known  within 
management  circles  for  his  Cynefin  Sensemaking  Framework,  states  that  his  framework, 
“allows  executives  to  see  things  from  new  viewpoints,  assimilate  complex  concepts,  and 
address  real-world  problems  and  opportunities.”94  The  literal  translation  of  Cynefin  is 
“habit  or  place.”  However,  Snowden  describes  the  word  as  signifying  “the  multiple 
factors  in  our  environment  and  our  experience  that  influence  us  in  ways  we  can  never 
understand.”95  The  idea  is  that  human  beings  are  in  and  of  themselves  a  living  and 
breathing  complexity  model  because  people  have  had  thousands  of  influencers  in  the  past 
but  can  never  really  know  how  those  influencers  are  impacting  them  at  any  given  time. 

As  illustrated  in  Figure  1,  the  Cynefin  framework  is  divided  into  four  quadrants 
with  a  fifth  domain  situated  in  the  middle.  In  the  Cynefin  Sensemaking  model,  the  data 
precedes  the  framework,  or  in  other  words,  the  framework  allows  patterns  to  emerge 
from  the  data.  Different  ways  of  thinking  and  acting  are  determined  and  allowed  based  on 
the  domain  occupied.  The  domains  include  the  simple,  complicated,  complex,  chaotic, 
and  disordered. 
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Figure  1.  Snowden’s  Cynefin  Framework96 


The  simple  domain  is  an  ordered  system  and  identified  by  problems  that  have  a  direct 
cause  and  effect  relationship.  In  this  domain,  the  protocol  is  to  sense  incoming  data, 
categorize  it,  and  respond.  Snowden  put  the  concept  of  best  practice  in  this  domain  to 
indicate  that  through  past  iterative  processes,  answers  can  be  known  in  advance  and  are 
characteristically  dependable.  In  this  domain,  new  learning  would  be  unnecessary 
because  results  are  highly  predictable.  In  essence,  data  comes  in,  is  quickly  categorized, 
and  the  response  (solution)  is  apparent  and  leads  to  an  expected  outcome. 

The  complicated  domain  is  also  an  ordered  system  and  characterized  by  a  process 
of  sensing,  analyzing,  and  responding.  The  real  difference  between  the  simple  domain 
and  the  complex  domain  is  basically  a  matter  of  time.  Data  is  easily  categorized  in  the 
simple  domain;  however,  data  in  the  complicated  domain  needs  further  analysis  before  a 
clear  solution  is  identified.  To  be  clear,  within  the  complicated  domain,  a  relationship  still 
exists  between  cause  and  effect  that  will  yield  a  solution.  However,  a  more  rigorous 
process  of  data  analysis  or  the  input  of  subject  matter  experts  is  necessary  to  come  up 
with  what  might  be  any  number  of  “right”  ways  of  solving  the  problem.  Thus,  as  a  way  to 
indicate  the  fact  that  several  ways  of  getting  to  a  solution  may  exist,  Snowden  labels 

96  Wikipedia,  “Cynefin,”  (n.d.),  http://en.wikipedia.org/wiki/Cynefm. 
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practice  in  this  domain  as  “good”  as  opposed  to  “best.”  Also,  in  this  domain,  knowledge 
and  infonnation  that  already  exists  is  sufficient  to  solve  the  problem;  thus,  traditional 
learning  models  (trial  and  error,  etc.)  are  unnecessary. 

The  complex  domain  is  unordered.  In  other  words,  cause  and  effect  is 
unpredictable  and  smart  and  best  practices  are  no  longer  reliable.  The  probe-sense- 
respond  method  promotes  what  Snowden  refers  to  as  “safe  to  fail”  experiments  that  lead 
to  amplification  if  the  experiment  succeeds  or  dampening  if  it  begins  to  fail.  In  this 
respect,  a  new  or  emergent  practice  will  develop,  and  in  this  sense,  emergent  practice 
arises  through  learning.  The  safe  to  fail  model  encourages  trying  new  things  to  manage 
the  situation.  Once  a  complex  space  is  detennined,  various  learning  strategies  and 
conditions  can  be  utilized;  some  of  which  are  described  in  the  next  chapter. 

Like  the  complex,  the  chaotic  is  also  an  unordered  domain  in  which  cause  and 
effect  relationships  cannot  be  determined.  Snowden  suggests  that  entering  this  domain 
deliberately  is  useful  for  innovation;  however,  if  it  is  entered  upon  accidentally,  the 
primary  objective  is  to  stabilize  by  using  an  act-sense-respond  method.  In  the  chaotic 
domain,  all  practice  will  be  novel  in  an  attempt  to  stabilize  the  situation  and  move 
towards  a  more  manageable  domain.  Finally,  Snowden  suggests  that  the  disordered 
domain  is  the  place  in  which  it  is  impossible  to  detennine  which  domain  is  occupied, 
although  according  to  Snowden,  it  is  also  the  domain  that  is  inhabited  most  of  the  time. 

The  following  chapter,  which  outlines  the  Emergency  Learning  Framework 
(ELF),  is  heavily  influenced  by  the  Cynefin  Framework.  In  essence,  the  researcher 
suggests  that  intracrisis  learning  is  unnecessary  within  the  ordered  domains  of  the 
framework.  Since  the  simple  domain  is  distinguished  by  best  practice,  pre-existing  plans, 
procedures  and  capabilities  should  be  immediately  accessible  to  solve  incoming 
problems.  In  other  words,  the  learning  has  already  occurred  and  solutions  are  predictable. 
Although  the  complicated  domain  has  room  for  more  than  one  right  answer,  it  is 
unnecessary  to  spend  precious  time  learning  in  this  domain  when  subject  matter  experts 
(SME)  are  available  to  solve  the  problem.  However,  the  researcher  proposes  that  it  is 
within  the  unordered  domains  that  real-time  learning  is  possible  within  an  emergency. 
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IV.  THE  EMERGENCY  LEARNING  FRAMEWORK  (ELF) 


A.  INTRODUCTION 

Chapter  IV  focuses  on  a  detailed  description  of  the  ELF.  The  ELF  is  broken  down 
in  to  six  individual  learning  frames: 

1 .  Intracrisis  Exercise — After  Action  Review  (AAR) 

2.  Intercrisis  Debrief — Exercise:  After  Exercise  Review  (AXR) 

3.  Intracrisis  Event — During  Event  Reflection  (DER) 

4.  Debrief-Event:  After  Event  Review  (AER) 

5.  Ongoing  Leaming/Planning  Meetings 

6.  Intercrisis  (or  Prescriptive)  Learning 

Each  frame  is  discussed  based  upon  theoretical  support  from  various  disciplines 
including  the  aforementioned  sensemaking  concepts,  individual  and  organizational 
learning  literature,  organizational  development  theory,  contemporary  psychology,  and 
recent  emergency  management  theory  and  practice.  Please  see  Appendix  B  for  a  visual 
representation  of  this  model. 

B.  FRAME  DISCUSSION 

1.  Intracrisis  Exercise — After  Action  Review  (AAR) 

a.  General  Discussion 

Depending  on  the  Cynefin  sensemaking  domain,  in  the  ELF,  the  revised 
AAR  concept  has  the  potential  to  be  a  form  of  intracrisis  learning  that  would  occur 
during  an  exercise.  Exercise  controllers,  evaluators,  as  well  as  the  incident  commander, 
and  a  learning  unit  leader,  would  have  the  ability  to  stop  an  exercise  at  any  time  to 
engage  in  this  type  of  real-time  learning.  They  might  do  this  to  review  capabilities, 
objectives,  actions  or  decisions  not  made  according  to  plan,  did  not  actuate  the  expected 
outcome,  or  most  importantly,  when  these  pieces  failed  to  identify  a  direct  course  of 
action  or  when  expected  capabilities  were  inadequate.  In  other  words,  in  the  language  of 
the  Cynefin  framework,  when  best  or  good  practice  is  unworkable  and  a  situation  has 
entered  into  one  of  the  unordered  domains. 
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To  be  clear,  this  type  of  review  would  happen  during  the  actual  exercise 
and  not  at  its  conclusion.  The  overall  goal  is  to  learn  to  stick  to  plans,  procedures,  and 
capabilities  when  they  are  effective,  but  also  to  abandon  them  quickly  when  they  do  not, 
and  in  turn,  engage  in  intracrisis  learning  methods  and  techniques  as  an  alternative.  In 
essence,  the  idea  is  to  use  a  learning  orientation  to  help  manage  uncertainty.97 

In  this  conception,  the  AAR  process  does  not  look  backward  to  gather 
lessons  learned  (and  implement  at  a  later  date)  but  becomes  an  opportunity  to  reflect, 
plan,  and  learn  in  real  time.  For  example,  Baird,  Holland,  and  Deacon  state  that  this  type 
of  AAR 

. .  .is  a  way  for  a  team  to  reflect  on  and  learn  while  it  is  perfonning.  Unlike 
other  project  post  mortems,  you  do  not  wait  until  the  patient  is  dead  to 
figure  out  what  went  wrong.98 

This  perspective  is  different  from  how  the  AAR  is  commonly  practiced  in  the  homeland 
security  enterprise.  In  an  AAR’s  current  conception,  controllers,  evaluators  or 
participants  normally  do  not  stop  an  exercise  to  have  a  team  reflect  on  real-time 
experiences  that  might  move  them  to  try  alternative  actions  resulting  in  a  better  or 
different  outcome.  Although  an  exercise  may  be  informally  interrupted  to  discuss  a 
capability,  objective  or  decision,  it  is  not  built  in  as  an  encouraged  disruptive  learning 
opportunity  to  help  participants  identify  new  knowledge,  practice  capabilities  or  solve 
real-time  problems. 

As  Baird  suggests,  the  AAR  as  currently  practiced  occurs  at  the  end  of  the 
exercise.  This  type  of  post-mortem  relies  on  a  system  of  capturing  knowledge  to  improve 
performance  in  the  future.  The  assumption  is  that  lessons  learned  in  one  experience  will 
be  applicable  to  the  next.  However,  no  guarantee  exists  that  what  has  been  learned  in  the 
past  will  be  of  value  in  the  future.  As  Choo  argues,  “While  retained  meanings  provide 


97  Moynihan,  “Learning  under  Uncertainty:  Networks  in  Crisis  Management,”  350. 

98  Baird,  Holland,  and  Deacon,  “Learning  from  Action:  Imbedding  More  Learning  into  the 
Performance  Fast  Enough  to  Make  a  Difference,”  22. 
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useful  guidance  from  the  past  in  order  to  start  action,  past  guidance  needs  to  be  tempered 
with  alertness  toward  the  unusual  and  unexpected.  Thus,  sense  making  is  also  a  constant 
‘struggle  for  alertness.’”99 

For  example,  Cynthia  Renaud  identifies  a  number  of  questions  that  an 
incident  commander  might  ask  in  the  initial  stages  of  a  complex  or  chaotic  incident  to 
help  in  the  struggle  to  stay  alert: 

•  What  has  happened  here? 

•  What  have  I  never  seen  before;  what  is  completely  foreign  to  me? 

•  What  have  I  seen  before;  what  is  familiar  to  me? 

•  What  do  I  know? 

•  What  do  I  need  to  know? 

Once  these  questions  are  answered,  the  incident  commander  can  then 

consider: 

•  What  do  I  want  to  do? 

•  What  do  I  have  to  do? 

•  What  can  I  do? 

Once  these  questions  are  answered,  an  order  emerges  from  the  chaos  and 
the  incident  commander  can  consider  the  last,  most  important  question: 

•  What  am  I  trying  to  accomplish  here?100 

Within  the  Cynefin  framework,  these  types  of  questions  could  be  asked 
when  an  incident  commander  is  found  within  an  unordered  domain  (chaotic  or  complex). 
Not  asking  these  types  of  questions,  and  relying  on  “lessons  learned”  from  past  events  (as 
encoded  in  plans  and  procedures)  or  even  currently  identified  capabilities  that  will  not 
solve  the  problem  could,  in  fact,  leave  responders  unable  to  truly  make  sense  of  a  current 
situation  that  might  demand  new  thinking  and  learning.  The  unique  nature  of  dynamic 


99  Cun  Wei  Choo,  The  Knowing  Organization  (New  York:  Oxford  University  Press,  2006),  85. 

100  Cynthia  Renaud,  “The  Missing  Piece  of  NIMS:  Teaching  Incident  Commanders  How  to  Function 
in  the  Edge  of  Chaos,”  Homeland  Security  Affairs  (2012):  5. 


37 


emergencies  requires  new  thinking  to  learn  and  adapt  to  ever  changing  situations. 
Relying  on  lessons  learned  from  previous  experiences  may  in  fact  delay  what  should  be  a 
novel  approach  to  a  novel  experience. 

However,  exercises,  if  designed  correctly,  should  create  unique,  unusual, 
and  dynamic  experiences  to  challenge  participants  to  leam  and  adapt  in  new  ways.  As 
suggested  earlier,  in  utilizing  the  Cynefin  framework,  these  types  of  experiences  fall  into 
what  Snowden  refers  to  as  the  unordered  domain  (complex  and  chaotic)  and  present  the 
need  for  an  emergent  learning  sensibility.  According  to  Williams,  Karousou,  and 
Mackness,  emergent  learning 

. . .  arises  out  of  the  interaction  between  a  number  of  people  and  resources, 
in  which  the  learners  organise  and  detennine  both  the  process  and  to  some 
extent  the  learning  destinations,  both  of  which  are  unpredictable.  The 
interaction  is  in  many  senses  self-organised,  but  it  nevertheless  requires 
some  constraint  and  structure.  It  may  include  virtual  or  physical  networks, 
or  both.101 

The  questions  identified  above  by  Renaud  could  help  a  team  to  self-organize  and  identify 
the  “process”  and  necessary  “learning  destinations.”  In  a  sense,  managing  emergent 
learning  opportunities,  tolerating  uncertainty  and  understanding  complexity,  become 
needed  homeland  security  capabilities  in  and  of  themselves.  This  idea  is  discussed  in 
more  detail  later  in  the  thesis. 

Garud,  Dunbar,  and  Bartel  describe  emergent  learning  as  something  that 
comes  from  unusual  experiences  and  in  turn 

...must  be  a  generative  process.  Specifically,  learning  must  occur  in  such 
a  way  that,  at  any  moment,  past  experiences  of  unusual  experiences  can  be 
mobilized  to  shape  but  not  determine  responses  to  current  and  future 
unusual  experiences.  For  such  a  process  to  unfold,  organizational  learning 
must  trigger  reflection  and  novel  action  rather  than  responses  detennined 
by  predefined  templates. 102 


101  Roy  Williams,  Regina  Karousou,  and  Jenny  Mackness,  “Emergent  Learning  and  Learning 
Ecologies  in  Web  2.0,”  International  Review  of  Research  in  Open  and  Distance  Learning,  2011, 
http://www.irrodl.org/index.php/irrodl/article/view/883/1686. 

102  Garud  Raghu,  Roger  L.  M.  Dunbar,  and  Caroline  A.  Bartel,  “Dealing  with  Unusual  Experiences:  A 
Narrative  Perspective  on  Organizational  Learning,”  Organization  Science  (2010):  1-2. 
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In  this  thesis,  Garud,  Dunbar,  and  Bartel  help  to  convey  the  point  that  learning  does  not 
need  to  occur  during  the  simple  or  even  complicated  domains  of  the  Cynefin  framework. 
In  these  domains,  pre-identified  capabilities,  plans  or  standard  operating  procedures 
(“predefined  templates”)  should  already  be  available  to  solve  incoming  problems.  If  these 
capabilities  or  documents  are  not  available,  SMEs  can  be  brought  in  to  help  solve  the 
problem.  In  these  cases,  best  or  good  practice  should  suffice.  Having  an  understanding 
that  within  the  simple  and  complicated  domains,  at  least  one  right  answer  exists  and  that 
x  will  (eventually)  equal  y  alleviates  the  necessity  to  engage  in  any  type  of  real-time 
learning  activity.  However,  Lieutenant  Joe  Moore  states  that, 

In  a  complex  situation,  most  of  what  you  learn  from  a  single  experience  is 
the  wrong  answer.  So  you  go  out  and  choose  a  different  answer  to  the 
problem,  and  it’s  wrong  too,  but  maybe  it’s  less  wrong.... You’ve  got  to 
learn  in  small  bites,  lots  of  them,  over  time,  and  they’ll  work,  eventually, 
into  a  complete  solution  to  the  problem.  This  cannot  be  accomplished  in  a 
one-time  reflection  event  that  happens  only  after  a  project  is  complete. 103 

Snowden’s  Cynefin  framework  helps  to  manage  this  type  of  “complex 
situation”  by  laying  out  the  instructions  (as  identified  in  the  complex  domain)  to  probe- 
sense-respond.  This  iterative  process  allows  for  a  type  of  experimentation  where  one 
engages  in  an  emergent  practice.  Within  this  practice,  results  that  are  sensed  to  be 
advantageous  are  amplified  while  results  that  are  not  working  are  dampened.  Regardless, 
cause  and  effect  in  this  type  of  situation  is  anything  but  predictable.  In  this  domain, 
practitioners  must  be  comfortable  with  periods  of  ambiguity  and  uncertainty  until  a 
solution  comes  into  focus. 

Charles  Lindblom  described  a  similar  type  of  process  as  “muddling 
through.”104  Ahlstrand,  Mintzberg,  and  Lampel  state  that  “Lindblom  suggested  that 
policy  making  (in  government)  is  not  a  neat,  orderly,  controlled  process,  but  a  messy  one 
in  which  policymakers  try  to  cope  with  a  world  they  know  is  too  complicated  for 


103  Lieutenant  Joe  Moore  in  Darling  and  Parry,  “After-Action  Reviews:  Linking  Reflection  and 
Planning  in  a  Learning  Practice,”  1. 

104  Charles  E.  Lindblom,  “The  Science  of  “Muddling  Through,”  Public  Administration  Review  19,  no. 
2  (Spring  1959):  79-88. 
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them.”105  This  type  of  “muddling  through”  suggests  a  certain  tolerance  for  ambiguity  and 
an  ability  to  let  go  of  control  and  make  mistakes.  Muddling  through  also  suggests  that 
response  leaders  are  aware  that  they  cannot  control  a  situation  but  must  work  with  it  to 
bring  it  eventually  into  a  more  manageable  domain. 

Donella  Meadows  describes  the  process  of  muddling  through  as  an 
opportunity  for  emergent  learning: 

The  thing  to  do,  when  you  don’t  know,  is  not  to  bluff  and  not  to  freeze, 
but  to  learn.  The  way  you  leam  is  by  experiment — or,  as  Buckminster 
Fuller  put  it,  by  trial  and  error,  error,  error.  In  a  world  of  complex  systems, 
it  is  not  appropriate  to  charge  forward  with  rigid,  undeviating  directives. 

“Stay  the  course”  is  only  a  good  idea  if  you’re  sure  you’re  on  course. 
Pretending  you’re  in  control  even  when  you  aren’t  is  a  recipe  not  only  for 
mistakes,  but  for  not  learning  from  mistakes.  What’s  appropriate  when 
you’re  learning  is  small  steps,  constant  monitoring,  and  a  willingness  to 
change  course  as  you  find  out  more  about  where  it’s  leading. 106 

Both  Meadows  and  Lieutenant  Moore  describe  a  process  of  learning  in 
small  bites,  experimentation,  trial  and  error,  and  failure.  The  process  speaks  to  constant 
adaptation,  and,  in  a  sense,  is  the  key  to  the  intracrisis  exercise  domain.  Within  the  ELF, 
exercise  AARs  should  be  used  to  develop  the  habit  of  an  organizational  orientation  to  an 
emergent  or  adaptive  learning  process  or  cycle.  In  his  now  classic  Experiential  Learning: 
Experience  as  the  Source  of  Learning  and  Development,  David  Kolb  states  that 

Learning  is  the  major  process  of  human  adaptation.  This  concept  of 
learning  is  considerably  broader  than  that  commonly  associated  with  the 
school  classroom.  It  occurs  in  all  human  settings,  from  schools  to  the 
workplace,  from  the  research  laboratory  to  the  management  board  room, 
in  personal  relationships  and  the  aisles  of  the  local  grocery... Therefore  it 
encompasses  other,  more  limited  adaptive  concepts  such  as  creativity, 
problem  solving,  decision  making,  and  attitude  change  that  focus  heavily 
on  one  or  another  basic  aspects  of  adaptation. 107 


105  Bruce  Ahlstrand,  Henry  Mintzberg,  and  Joseph  Lampel,  Strategy  Safari  (Simon  &  Schuster,  Inc., 
February  21,  2001),  Kindle  edition,  loc.  2567-2574. 

10®  Donella  Meadows,  Thinking  in  Systems:  A  Primer  (Chelsea  Green  Publishing,  2011),  Kindle 
edition,  loc.  3409-3414. 

107  David  Kolb,  Experiential  Learning:  Experience  as  the  Source  of  Learning  and  Development 
(Englewood  Cliffs,  NJ:  Prentice  Hall,  1984),  32. 
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Exercise  AARs  afford  the  luxury  of  being  able  to  stop  the  action  and  utilize  this 
experiential  learning  process.  In  the  language  of  David  Snowden,  the  intracrisis  exercise 
environment  is  inherently  safe  to  fail.  According  to  Snowden: 

Safe-fail  Probes  are  small-scale  experiments  that  approach  issues  from 
different  angles,  in  small  and  safe-to-fail  ways,  the  intent  of  which  is  to 
approach  issues  in  small,  contained  ways  to  allow  emergent  possibilities  to 
become  more  visible.  The  emphasis,  then,  is  not  on  ensuring  success  or 
avoiding  failure,  but  in  allowing  ideas  that  are  not  useful  to  fail  in  small, 
contained  and  tolerable  ways.  The  ideas  that  do  produce  observable 
benefits  can  then  be  adopted  and  amplified  when  the  complex  system  has 
shown  the  appropriate  response  to  its  stimulus.108 

This  type  of  experimental  learning  is  similar  to  Kolb’s  experiential  learning  cycle.  As  can 
be  seen  from  Figure  2,  Kolb  identifies  a  four-fold  cycle  of  experience-reflection- 
conceptualization  and  experimentation.  The  cycle  also  contains  elements  of  Renaud’s 
mental  processing  questions.  To  be  clear,  this  cycle  is  relevant  to  Snowden’s  unordered 
domains  where  documented  capabilities,  plans,  and  SOPs  are  not  available  and  solutions 
must  emerge. 


108  David  Snowden,  “Safe-to-Fail  Probes,”  CognitiveEdge  Network,  (n.d.),  http://cognitive- 
edge.com/library/methods/safe-to-fail-probes/. 
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Kolb’s  cycle  is  relevant  to  situations  in  which  best  practices  (cause  and  effect)  in  the 
shape  of  tested  plans  and  procedures  are  unknown  and  an  experimental  orientation  is 
necessary.  In  both  safe  to  fail  and  experiential  learning,  solutions  emerge  through  a 
formalized  and  expedited  process  of  “muddling  through.”  However,  as  suggested  earlier, 
a  necessary  condition  for  this  type  of  learning  is  a  tolerance  of  ambiguity  and  uncertainty 
as  events  unfold  and  understanding  of  the  new  reality  starts  to  become  clearer.  As  Donald 
Schon,  an  influential  thinker  on  reflective  practice,  states: 

The  practitioner  allows  himself  to  experience  surprise,  puzzlement,  or 
confusion  in  a  situation  which  he  finds  uncertain  or  unique.  He  reflects  on 
the  phenomenon  before  him,  and  on  the  prior  understandings  which  have 
been  implicit  in  his  behaviour.  He  carries  out  an  experiment  which  serves 
to  generate  both  a  new  understanding  of  the  phenomenon  and  a  change  in 
the  situation.110 


109  Tim  Goodenough,  “Learning  to  Learn:  The  Unsexy  Edge  in  Sport,”  Coaching  Unity  International, 
September  27,  2008,  http://www.coachingunity.co.za/learning21earn.html. 

1 10  Donald  Schon,  Tlie  Reflective  Practitioner:  How  Professionals  Think  in  Action  (New  York:  Basic 
Books,  1983),  68. 
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It  must  be  noted  that  tolerating  this  type  of  confusion  and  uncertainty  may 
be  one  of  the  biggest  hurdles  in  the  learning  process.  To  reach  this  place  of  learning,  Don 
Michael  states  that  a  society  would  have  to 

...self-consciously  question  its  premises  as  well  as  its  actions,  accept 
uncertainty,  and  measure  leadership  competence  at  all  levels  by  its 
capacity  to  acknowledge  ignorance  and  uncertainty  as  prerequisites  for 
discovery  and  change,  i.e.  as  the  conditions  for  learning. 111 

Obviously,  this  is  a  tall  order.  However,  as  a  condition  for  learning,  what  other  way  can 
one  begin  the  process  but  by  acknowledging  what  they  do  not  know?  Kees  van  der 
Heijden  states  it  this  way: 

In  a  situation  of  uncertainty  planning  becomes  learning,  which  never 
stops.  We  have  an  in-built  urge  to  try  to  pin  down  situations,  and  to  try  to 
reach  a  point  where  we  have  got  it  sorted  out  in  our  mind.  If  uncertainty  is 
acknowledged  it  is  not  any  longer  possible  to  take  this  position  of  “we 
have  done  the  planning. 1 12 

Unfortunately,  in  the  homeland  security  enterprise,  practitioners  are  in  the 
precarious  position  of  being  expected  to  know,  or  of  having  to  have  “done  the  planning,” 
for  if  they  do  not  know  or  have  not  planned,  how  can  U.S.  citizens  (and  responders 
themselves)  have  confidence  that  they  are  secure?  The  trick  is  to  move  into  a  mindset  of 
continual  learning  because  as  Moynihan  states  “Learning  helps  to  manage  uncertainty”113 

b.  Conclusion 

Ultimately,  the  problem  with  the  current  AAR  model  is  that  by  waiting 
until  the  end  of  an  exercise,  planners  and  participants  are  forfeiting  multiple  reflective 
learning  opportunities.  By  waiting  until  the  exercise  is  over,  responders  solve  problems 
that  existed  in  the  past  instead  of  learning  how  to  think  critically  and  create  solutions  in 
the  present.  Unless  these  solutions  can  be  identified  as  a  best  or  good  practice,  a 
possibility  always  exists  that  codifying  a  solution  that  worked  at  one  point  and  time  may 
not  necessarily  work  in  the  next. 

111  Michael,  “In  Search  of  the  Missing  Elephant:  Selected  Essays,”  17. 

1 12  Kees  van  der  Heijden,  Scenarios:  The  Art  of  Strategic  Conversation ,  Kindle  edition,  16. 

113  Moynihan,  “Learning  under  Uncertainty:  Networks  in  Crisis  Management,”  350. 
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By  using  the  Cyncfin  framework  to  identify  both  ordered  and  unordered 
domains  up  front,  responders  may  be  able  to  save  precious  time  by  either  identifying  best 
or  good  practice  solutions,  or  enter  into  a  more  reflective  and  experimental  learning 
mindset.  In  this  respect,  the  work  in  unordered  domains  becomes  more  process-oriented 
as  opposed  to  the  more  predictable  solution-centered  work  that  occurs  in  more  ordered 
domains. 

Practicing  these  types  of  reflective  techniques  during  an  exercise  may 
make  it  possible  that  these  types  of  learning  habits  and  ways  of  thinking  could  cross  over 
into  real  emergency  response.  In  doing  so,  it  is  possible  to  address  deficiencies  in  the 
current  approach,  such  as  assuming  subsequent  events  will  mirror  past  events  without 
unusual  or  unexpected  turns  or  relying  on  Taleb’s  narrative  fallacy. 

2.  Intercrisis  Debrief — Exercise:  After  Exercise  Review  (AXR) 

a.  General  Discussion 

The  next  domain  in  the  ELF  is  the  AXR.  To  be  clear,  the  AXR  is  now  in 
the  place  formally  occupied  by  the  AAR.  As  discussed  earlier,  in  the  language  of  HSEEP, 
this  session  is  called  a  Hot  Wash  and  allows  for  player  self-assessment  and  also  gives 
evaluators  an  opportunity  to  clarify  points  and  collect  any  missing  infonnation  regarding 
tested  capabilities.  All  this  data  is  then  returned  to  an  evaluation  team  where  it  is 
analyzed  and  transformed  “...into  narratives  that  address  the  course  of  exercise  play, 
demonstrated  strengths,  and  areas  for  improvement.”114 

As  suggested,  like  the  previous  reliance  on  plans  and  procedures,  a 
capabilities  model  may  have  utility  within  an  ordered  domain;  however,  specific 
capabilities  never  considered  or  identified  may  in  fact  be  the  ones  most  needed  in  a 
complex  or  chaotic  situation.  This  type  of  strategy  (reliance  on  predetermined  plans  or 
abilities)  is  precisely  what  reveals  what  possibly  may  be  a  responder’s  most  necessary 
capability;  the  one  that  makes  it  possible  to  make  sense  of,  adapt  and  learn  within 


114 


Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP),  Volume  III,  15. 
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unordered  domains.  Being  able  to  quickly  move  away  from  established  plans,  procedures, 
and  capabilities,  and  utilize  reflective  learning  tools,  is  a  capability  that  FEMA  has  not 
yet  identified. 

One  way  this  strategy  could  be  practiced  during  the  AXR  is  to  use  the 
Cynclin  framework  explicitly  to  identify  ordered  and  unordered  learning  practices  once 
the  exercise  is  complete.  Thus,  instead  of  just  reducing  the  activity  to  an  assessment  of 
how  well  predetennined  capabilities  were  performed  or  trying  to  decipher  what  worked 
and  what  did  not,  the  AXR  could  also  be  used  to  identify  when  predetermined 
capabilities,  plans,  and  procedures  were  sufficient  to  manage  an  incoming  problem 
(ordered  domain)  as  opposed  to  when  the  data  suggested  new  learning  was  necessary 
(unordered  domain).  In  performing  this  type  of  activity  after  the  exercise,  responders 
could  possibly  become  more  adept  at  perfonning  this  type  of  reflection  in  real  time  to 
help  make  sense,  and  in  turn,  detennine  if  ordered  learning  tactics  (referring  to  known 
plans,  procedures  and  capabilities)  are  sufficient  or  if  the  incoming  data  suggests  that 
unordered  emergent  tactics  (i.e.,  probe-sense-respond,  reflective  questioning)  would  be 
more  helpful. 


However,  HSEEP  Volume  III  does  allude  to  another  necessary  component 
of  the  AXR.  Once  the  evaluators  collect  all  the  data 

...it  is  critical  for  [them]  to  discover  not  only  what  happened,  but  why 
events  happened.  Each  task  that  is  not  completed  as  expected  offers 
evaluators  the  opportunity  to  search  for  a  root  cause.  A  root  cause  is  the 
source  of  or  underlying  reason  behind  an  identified  issue  (as  uncovered 
during  careful  analysis)  toward  which  the  evaluator  can  direct  an 
improvement.  To  arrive  at  a  root  cause,  an  evaluator  should  attempt  to 
trace  the  origin  of  each  event  back  to  earlier  events  and  their  respective 
causes.  Root  cause  analysis  may  also  require  the  review  and  evaluation  of 
an  entity’s  emergency  plans;  training  programs;  and  other  plans,  policies, 
and  procedures.115 

In  this  passage,  the  authors  of  HSEEP  seem  to  make  a  direct  reference  to  Chris  Argyris 
and  Donald  Schon  and  their  work  on  single  and  double  loop  learning.116  Argyris  and 


115  Homeland  Security  Exercise  and  Evaluation  Program  (HSEEP),  Volume  III,  15. 

116  Argyris  and  Schon,  Organizational  learning:  A  Theory’  of  Action  Perspective. 
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Schon  contend  that  most  of  what  is  called  learning  is  single  loop.  This  type  of  learning  is 
basically  using  behaviors  and  tactics  to  solve  problems  in  the  real  world.  Single  loop 
learning  is  focused  on  fixing  the  symptoms  of  the  problem  and  improving  the  system  as  it 
exists.  As  the  authors  of  HSEEP  suggest,  it  is  necessary  to  go  beyond  this  type  of  single 
loop  learning,  find  a  root  cause,  and  detennine  its  origin  so  that  the  problem  can  be 
addressed  at  a  deeper  organizational  level.  However,  the  authors  of  HSEEP  do  not  go  far 
enough.  Their  suggestion  that  a  root  cause  analysis  only  considers  “an  entity’s  emergency 
plans;  training  programs;  and  other  plans,  policies,  and  procedures”  does  not  consider  the 
overall  organizational  culture. 

As  can  be  seen  from  Figure  3,  double  loop  learning  considers  an 
organization’s  values  and  beliefs,  as  well  as  its  underlying  assumptions.  In  this  respect, 
double  loop  learning  pushes  back  on  and  challenges  existing  organizational  cultural. 
Beyond  just  plans,  policies,  procedures,  and  capabilities,  the  double  loop  model  asks  an 
organization  to  consider: 

...organizational  inquiry  which  resolve  incompatible  organizational  nonns 
by  setting  new  priorities  and  weightings  of  nonns,  or  by  restructuring  the 
norms  themselves  together  with  associated  strategies  and  assumptions.117 


117  Argyris  and  Schon,  Organizational  learning:  A  Theory’  of  Action  Perspective,  18. 


46 


Single-Loop  Learning 

The  most  common  style  of  learning 
is  just  problem  solving  -  improving 
the  system  as  it  exists. 


r 

1 

r  ^ 

r 

Assumptions 

Why  We  Do  What  We  Do 

Strategies  and 
Techniques 

What  We  Do 

Results 

What  We  Get 

_ A 

L _ A 

SB 

A 

Double-Loop  Learning 

More  than  just  fixing  the  problem,  this 
style  of  learning  questions  the 
underlying  assumptions,  values  and 
beliefs  behind  what  we  do. 


Figure  3 .  Double  Loop  Learning 1 1 8 


During  an  AXR  (and  subsequent  leaming/planning  meetings — described  later),  an  agency 
could  reflect  on  the  single  loop  learning  that  occurred  during  the  exercise  and  discuss  the 
larger  associated  cultural  issues  that  might  be  needed  to  truly  affect  the  root  cause  that 
might  be  enforcing  it.  Documentation  of  this  type  of  distinguishing  process  (single  vs. 
double)  could  also  help  an  agency  detect  patterns  over  time  that  could  point  to  truly 
entrenched  values  and  norms  that  might  be  difficult  to  uncover  during  just  one  session. 
Analysis  of  this  type  of  longer  term  data  collection,  beyond  just  collecting  information  on 
what  happened  during  the  exercise,  could  act  as  a  type  of  meta-leaming  reflection 
complimenting  the  reflective  practices  discussed  earlier  in  this  section. 

b.  Conclusion 

Although  HSEEP  promotes  the  use  of  root  cause  to  uncover  deeper 
solutions,  the  problem  is  that  these  fixes  are  only  single  loop  in  nature.  Response 
organizations  must  dig  deeper  and  uncover  more  culturally  entrenched  assumptions, 
norms,  values,  etc.  that  might  be  hindering  real  organizational  learning  and  change.  These 

1 '8  Andrew  Bryant,  “Reflecting  and  Learning:  2009  to  2010,”  The  self  leadership  blog,  entry  posted 
December  4,  2009,  http://www.selfleader.com/blog/coaching/reflecting-and-leaning-2009-to-2010/. 
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double  loop  fixes  allow  responders  to  identify  where  a  discrepancy  exists  between  values 
and  practice  to  align  organizational  beliefs  and  values  with  their  planning  and  actions. 

3.  Intracrisis  Event — During  Event  Reflection  (DER) 

a.  General  Discussion 

Obviously,  an  actual  emergency  cannot  be  interrupted  to  reflect  on 
specific  objectives,  goals,  and  capabilities  as  a  way  to  engage  in  an  emergent  learning 
process.  However,  a  number  of  ways  that  this  type  of  reflective  process  can  still  be 
accomplished  is  to  facilitate  and  encourage  learning  during  a  real  emergency.  To  be  clear, 
the  underlying  motive  of  the  entire  framework  is  to  infuse  a  continuous  learning 
sensibility  into  a  response  agency  by  adopting  a  philosophy  of  continual  learning  though 
reflection  and  action.  In  other  words,  a  shift  in  organizational  culture  and  values  is 
necessary. 

For  example,  within  an  emergency  coordination  center  (ECC),  it  is 
customary  to  engage  in  a  shift  briefing  (fonnal  handoff  between  incoming  and  outgoing 
responders)  at  the  beginning  of  each  operational  period.  According  to  the  FEMA  website 
the,  “briefing  is  conducted  at  the  beginning  of  each  Operational  Period  and  presents  the 
Incident  Action  Plan  [IAP]  to  supervisors  of  tactical  resources.”119  The  shift  briefing  is  a 
logical  point  within  the  operational  period  to  make  the  connection  between  planning  and 
learning. 

Kees  van  der  Heijden  clearly  identifies  this  connection  by  stating, 
“Learning  can  take  place  only  if  experience  deviates  from  the  plan  in  an  unexpected  way. 
If  everything  happens  according  to  expectation  there  is  no  learning.”120  This  point  is 
crucial.  Seasoned  emergency  managers  understand  that  existing  plans,  procedures,  and 
capabilities  will  not  always  be  sufficient  to  manage  an  emergency.  Within  the  Cynefin 
Framework,  at  this  time,  the  data  will  suggest  that  a  response  is  in  an  unordered  domain. 


119  FEMA,  Natural  Preparedness  Directorate,  “Planning  Process,”  (n.d.), 
http://training.fema.gov/EMIWeb/IS/ICSResource/assets/PlanningP.pdf,  1. 

120  van  der  Heijden,  Scenarios:  The  Art  of  Strategic  Conversation ,  44. 
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When  best  and  good  practice  can  no  longer  suffice,  emergent  practice  will  have  to  be 
employed.  Van  der  Heijden  describes  emergence, 

...as  the  appearance  of  unpredictable  or  incalculable  behaviour  resulting 
from  the  interaction  of  many  simple  components  that  cannot  be  derived 
from  knowledge  of  what  each  component  of  the  system  does  in 
isolation. 121 

In  the  language  of  the  Cynefin  Framework,  a  responder  in  these  types  of 
unordered  domains  would  use  a  probe/act-sense-respond  learning  strategy.  By  taking  on  a 
probing  or  acting  sensibility  within  an  unordered  environment,  a  type  of  experimentation 
occurs  that  would  allow  for  amplifying  or  dampening  results  until  some  type  of  order 
(coherent  patterns  within  the  environment)  becomes  visible.  As  Snowden  states,  “Leaders 
who  try  to  impose  order  in  a  complex  context  will  fail,  but  those  who  set  the  stage,  step 
back  a  bit,  allow  patterns  to  emerge,  and  determine  which  ones  are  desirable  will 
succeed.”122 


Thus,  an  IAP  could  be  assessed  using  the  Cynefin  framework.  If  predefined 
tactics  were  actuating  the  expected  results,  the  plan  would  not  need  to  be  altered.  The 
data  would  suggest  that  the  responders  are  in  an  ordered  domain  that  is  understood  and 
able  to  be  managed  by  preexisting  knowledge  and  actions  (plans,  procedures,  capabilities, 
etc.).  However,  if  IAP  tactics  were  not  bringing  about  expected  results,  a 
leaming/planning  strategy  could  be  invoked  to  “allow  patterns  to  emerge”  and  bring  a 
sense  of  order  back  to  the  environment.  As  van  der  Heijden,  referencing  both  Kolb  and 
Weick  states: 

And  so  we  learn!  Our  new  reflection  on  the  results  of  our  actions  shows  us 
that  our  theory  needs  developing  again.  And  the  process  continues. 
Fundamental  in  the  Kolb  model  is  the  role  of  action  in  any  learning.  The 
importance  of  the  inclusion  of  action  in  the  learning  process  was 
effectively  summarized  by  Weick  when  he  suggested  “how  do  I  know 
what  I  think  until...  I  see  how  I  act?”(Weick,  1979) 


121  van  der  Heijden,  Scenarios:  Tlte  Art  of  Strategic  Conversation,  33. 

122  Mary  E.  Boone  and  David  J  Snowden,  “A  Leaders  Framework,”  Harvard  Business  Review’ 
November  5,  2007. 
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Applied  to  strategy  development  the  learning  model  projects  integration  of 
experience,  sense-making  and  action  into  one  holistic  phenomenon.  It 
suggests  that  for  things  to  go  right  we  need  to: 

•  Perceive  weak  signals, 

•  Remember  lessons  from  the  past,  and 

•  Adapt  quickly  if  these  have  not  prepared  us  for  what 
actually  happens. 

The  model  is  based  on  the  idea  of  continuous  development  and 
improvement,  rather  than  “the  one  right  answer.”  Therefore,  the  process 
view  requires  less  emphasis  on  forecasts.  Instead  uncertainty  and 
ambiguity  are  faced  head  on. 123 

However,  how  does  one  meet  uncertainty  and  ambiguity  head  on 
especially  when  doing  so  might  lead  to  unwanted  results?  Donald  Michael  stresses  that 
“Leaders,  at  whatever  level,  cannot  acknowledge  publicly,  or  often  even  to  themselves, 
either  their  ignorance  or  impotence.  And  if  they  should  do  so,  citizens  would  replace 
them  with  others  who  claim  they  can  do  what  needs  to  be  done.”?124  In  other  words,  a 
leader  who  acknowledges  this  type  of  vulnerability  may  suffer  a  political  cost. 

However,  this  reality  is  unfortunate.  Although  vulnerability  within  the 
realm  of  emergency  management  might  cause  a  political  death,  at  least  anecdotal 
evidence  exists  that  this  trait  can  be  a  powerful  identifier  of  solid  leadership.  Russell 
Schweiss,  quoted  in  Graeme  Thomson’s  article  entitled,  “The  Art  and  Science  of 
Experiential  Leadership:  Culture  at  the  Core  of  Process  Change  Success,’  states, 
“vulnerability  and  humility  are  watennarks  of  good  leadership.  Sometimes  there’s 
nothing  healthier  than  being  vulnerable  in  front  of  people  that  you’re  working  with 
regularly,  so  they  can  see  your  humanity.  Because  they  are  human,  too,  they  know  they 
have  weaknesses.  To  see  someone  else  admit  to  it  gives  them  the  freedom  to  do  the 
same.”125  Bunker  states  that,  “Expressing  vulnerability  becomes  a  leadership  tool  when  it 


123  van  der  Heijden,  Scenarios:  The  Art  of  Strategic  Conversation,  37-39. 

124  Donald  N.  Michael,  Learning  to  Plan  and  Planning  to  Learn  (St.  Michaels,  MD:  Miles  River 
Press,  1973),  23. 

125  Russell  Schweiss,  quoted  in  Graeme  Thomson,  “The  Art  and  Science  of  Experiential  Leadership: 
Culture  at  the  Core  of  Process  Change,”  Journal  of  Business  Strategy  31,  no.  4  (2010):  89. 
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opens  the  door  to  connecting  with  others  at  the  basic  level  of  humanness.”126  In  a  sense, 
vulnerability  has  the  potential  to  convey  to  others  that  one  does  not  know  or  is  uncertain. 
In  this  respect,  vulnerability  becomes  a  precursor  for  learning  because  it  is  only  possible 
to  learn  when  something  is  unknown.  In  other  words,  learning  cannot  occur  without  some 
sort  of  vulnerability.  However,  even  though  an  incident  commander  may  be  willing  to  be 
vulnerable  in  front  of  a  response  team,  the  extra  capacity  needed  to  do  the  higher  level 
learning  work  may  not  be  available  as  a  capability. 

In  these  instances,  it  may  take  more  than  critically  reflecting  upon  an  IAP 
during  a  shift  change.  With  this  understanding  in  mind,  the  Electricite  de  France,  the 
second  largest  electric  utility  in  the  world,  has  devised  a  new  reflective  learning  tool:  The 
Rapid  Reflection  Force  (RRF).  Simply  put,  the  RRF  is  an  interdisciplinary  group  charged 
with  assisting  the  incident  commander  “grasp  and  confront  issues  raised  by 
unconventional  situations.”127  Fagadec  describes  the  RRF  as 

...a  spur  that  will  prod  crisis  leadership  to  keep  moving,  keep  thinking, 
never  indulging  in  trench  warfare  against  unconventional  disruptions — as 
such  events  will  instantly  overwhelm  or  turn  round  all  attempts  to  draw 
static  lines  of  defense  or  restore  intellectual  comfort  zones.  With  this 
objective  in  mind,  the  critical  weapon  in  the  RRF’s  arsenal  turns  out  to  be 
insightful  questions,  rather  than  prefonnatted  answers,  which  are  the 
building  blocks  of  artificial  certainty,  the  Trojan  horses  of  instant 
collapse.128 

In  this  sense,  the  RRF  is  much  more  than  a  tool  to  use  during  shift  changes  or  even  when 
able  actually  to  stop  an  exercise  to  reflect  and  leam.  The  RRF  allows  for  a  type  of  rneta- 
leaming  as  a  shadow  team  reflects  on  real-time  operational  issues  that  occur  in  unordered 
domains.  Fagadec  and  Topper  state  that: 


126  Kerry  A.  Bunker,  “The  Power  of  Vulnerability  in  Contemporary  Leadership,”  Consulting 
Psychology’  Journal  49,  no.  2  (1997):  134. 

127  Pierre  Beroux,  Xavier  Guilhou,  and  Patrick  Lagadec,  “Rapid  Reflection  Forces  Put  to  the  Reality 
Test,”  Crisis  Response  4,  no.  2  (2008):  38. 

128  Project  White  Horse  084640,  “DaVinci’s  Horse  #6,  Part  2,”  (n.d.), 
http://www.projectwhitehorse.com/pdfs/Davincis6%20Part_%202Final.pdf. 
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Along  with  the  more  “tactical”  crisis  teams,  focused  entirely  on  immediate 
operational  responses,  plans  and  logistics,  such  RRF  teams  promptly 
undertake  four  broad  lines  of  interlinked  questioning  processes. 


•  What  is  the  essence  of  the  problem?  It  is  crucial  to 
understand  the  nature  of  the  crisis,  and  to  anticipate  the 
possible  mutations  of  the  challenges. 

•  What  are  the  major  pitfalls?  When  the  pressure  of  events 
becomes  extreme,  a  very  nonnal  tendency  is  to  become 
mired  in  the  most  counterproductive  ruts.  It  is  crucial, 
immediately,  to  think  about  the  major  errors  to  avoid.  And 
the  first  is  a  wrong  framing  of  the  issue. 

•  What  is  the  map  of  actors;  what  networks  are  needed? 
Extreme  crises  strike  at  the  system  in  ways  that  are  hard  to 
anticipate,  and  involve  complex  emerging  networks  of 
actors.  Mapping  those  networks  is  crucial. 

•  What  constructive  initiatives  can  the  RRF  suggest?  The 
most  important  thing  is  not  to  pore  over  statistical  lists  or  to 
compile  all  the  infonnation  possible,  but  rather  to  try  to 
discern  one  or  a  few  critical  initiatives  that  could  introduce 
a  new  set  of  rules. 129 

Once  again,  the  work  of  Snowden,  Kolb  and  Renaud  comes  to  mind.  However,  in  this 
instance,  a  separate  team  is  used  to  provide  a  different  type  of  consciousness  and 
perspective  that  allows  the  responding  team’s  decisions  and  actions  to  be  observed  by  a 
theoretically  less  attached  and  distracted  reflective  unit.  This  type  of  unit  might  be  called 
in  once  an  emergency  has  been  determined  to  exist  within  an  unordered  domain  in  which 
learning  in  real  time  would  be  necessary  because  existing  plans,  procedures  and 
capabilities  were  not  working. 

Beroux,  Guilhou,  and  Lagadec  summarize  the  promise  of  the  RFF  as  a: 

...new  instrument  to  begin  charting  emerging  risks  and  crises,  and  the 
appropriate  responses.  This  is  because  it  focuses  on  questions,  on 
creativity,  rather  than  on  ready-made  answers.  It  calls  for,  and  elicits,  the 
sharing  of  questions,  intuitions,  and  open-minded  approaches.  It 


l29  Patrick  Lagadec  and  Benjamin  Topper,  “How  Crises  Model  the  Modern  World,”  Journal  of  Risk 
Analysis  and  Crisis  Response  2,  no.  1  (May  2012):  28. 
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concentrates  on  flagging  specific  ways  out,  not  on  the  absurd  ambition  to 
develop  global,  final  answers.  Those  are  no  longer  attainable  -  if  they  ever 
were  -  in  today’s  chaotic  environment. 130 


This  type  of  tool  is  another  intriguing  option  for  use  within  unordered  domains.  To  be 
clear,  the  “ready-made”  answers  of  existing  protocol,  plans,  and  capabilities  have  use 
within  the  ordered  domains  as  long  as  they  continue  to  yield  the  expected  results. 
However,  having  an  outside  group  that  focuses  on  questions  when  the  answers  no  longer 
work  suggests  a  higher  order  of  consciousness  dearly  needed  to  allow  for  a  new  way  of 
thinking  about  the  crisis  when  existing  templates  and  ready  made  answers  do  not  work. 

van  der  Heijden  poses  the  question  that  the  RFF  might  solve  in  this  way: 

. .  .how  we  can  create  the  conditions  for  true  strategic  creativity  to  emerge. 

Pierre  Wack  suggested  that  if  we  look  long  and  hard  enough  the  moment 
of  reframing  will  always  come,  when  we  suddenly  see  the  world  in  a  new 
light,  and  gain  a  unique  insight  in  how  to  find/regain  success.  An  example 
of  management  intervention  for  this  purpose  is  the  creation  of  more 
“space”  for  the  infonnal  conversation,  by  creating  a  process  of  events 
through  which  views  can  be  exchanged  outside  the  pressure  of  immediate 
decision  making.  Taking  the  strategic  conversation  away  from  the  pressure 
of  immediate  decisions  allows  people  to  explore  possibilities  more 
freely. 131 

In  a  sense,  the  RFF  functions  as  a  secondary  system  ready  to  engage  in  this  type  of 
strategic  conversation  when  the  first  response  system  is  no  longer  adequate.  The  RFF  is 
the  space  needed  to  engage  a  necessary  refraining  to  explore  alternate  possibilities.  This 
type  of  dual  operation  reflects  the  ideas  in  Daniel  Kahneman’s  recent  book  Thinking , 
Fast  and  Slow . 132  Kahneman  describes  how  research  has  suggested  that  humans  rely  on 
the  interaction  of  two  different  internal  systems  to  help  people  live  in  and  make  sense  of 
the  world.  In  normal  situations,  system  one  runs  automatically  while  system  two  is  in  the 
background  at  a  low  level  of  engagement.  However,  when  things  become  difficult  for 
system  one,  a  type  of  override  occurs  and  system  two  is  activated.  As  Kahneman  states: 


130  Beroux,  Guilhou,  and  Lagadec,  “Rapid  Reflection  Forces  Put  to  the  Reality  Test,”  40. 

131  van  der  Heijden,  Scenarios:  Tite  Art  of  Strategic  Conversation,  xvii. 

I3-  Daniel  Kahneman,  Thinking,  Fast  and  Slow  (Macmillan,  2011),  Kindle  edition. 
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When  System  1  runs  into  difficulty,  it  calls  on  System  2  to  support  more 
detailed  and  specific  processing  that  may  solve  the  problem  of  the 
moment.  System  2  is  mobilized  when  a  question  arises  for  which  System  1 
does  not  offer  an  answer. . .System  2  is  activated  when  an  event  is  detected 
that  violates  the  model  of  the  world  that  System  1  maintains. 133 

With  respect  to  the  Cynefin  framework,  system  two  is  activated  when  the  data  suggests 
that  an  unordered  domain  has  been  encountered  (a  violation  of  system  1).  In  a  complex  or 
chaotic  domain,  the  RFF  becomes  a  type  of  system  two  function  that  helps  generate  the 
questions  and  probes  that  can  provide  an  emergent  solution  when  a  clear  answer  (current 
plans,  procedures  and  capabilities)  is  not  available.  The  RFF  (or  system  2)  can  engage  in 
the  types  of  reflective  questioning  and  subsequent  actions  that  Snowden,  Kolb,  and 
Renaud  encourage.  Early  anecdotal  data  suggests  that  the  RFF  can  be  a  useful  tool.  As 
Beroux,  Guilhou,  and  Lagadec  state: 

Real  life  incidents  and  exercises  have  shown  that  the  RRF  can  genuinely 
become  a  pillar  of  strength  around  which  an  organization  can  coalesce. 

The  RRF  can  benefit  all.  On  a  global  scale,  it  can  help  an  entire 
organization  develop  strength,  coherence,  stability,  and  strategic 
intelligence,  and  thereby  address  the  most  difficult  -  and  increasingly 
frequent  -  challenges  of  our  turbulent  times.  The  RRF  is  also  a  steady 
driver  for  benchmarking,  partnerships,  and  shared  initiatives.134 

b.  Conclusion 

A  problem  for  response  leaders  is  that  is  difficult  to  lead  and  engage  in  a 
real-time  event  while  simultaneously  being  able  to  reflect  and  leam.  The  use  of  the 
Cynefin  framework  (or  other  reflective  tools)  during  operational  shift  changes  can  help  to 
gauge  how  well  actions  are  helping  to  move  from  the  unordered  domains  to  ordered 
domains.  These  practices  can  provide  some  perspective  on  how  well  the  IAP  is  working 
and  if  certain  elements  need  to  be  dampened  or  amplified. 


133  Kahneman,  Thinking,  Fast  and  Slow,  Kindle  edition,  loc.  435-438. 

134  Beroux,  Guilhou,  and  Lagadec,  “Rapid  Reflection  Forces  Put  to  the  Reality  Test,”  40. 
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Another  possible  solution  is  the  RRF.  This  instrument  can  be  used  as  a 
type  of  System  2  backup  when  events  are  unpredictable  and  ambiguous.  Having  a 
secondary  team  to  assist  in  sense  making,  systems  analysis,  networking,  and  alternative 
questioning  provides  response  leaders  with  a  differing  perspective  that  may  help  identify 
new  plans  and  solutions. 

Expecting  response  leaders  to  both  lead  and  reflect  during  complex  and 
chaotic  events  might  be  asking  too  much.  However,  having  a  separate  team  that  could 
manage  these  necessary  functions  could  help  meet  the  unexpected  challenges  inherent  in 
any  crisis. 

4.  Debrief  Event — After  Event  Review  (AER) 

a.  General  Discussion 

As  the  AXR  discussed  earlier  in  this  chapter,  the  AER  would  be 
configured  as  a  type  of  hot  wash  that  would  occur  directly  after  the  event  had  formally 
ended.  Like  the  AXR,  the  purpose  of  this  meeting  would  be  focused  on  identifying  single 
and  double  loop  learning  by  utilizing  the  Cynefin  framework  to  track  how  plans  and 
procedures  moved  through  the  domains  as  well  for  a  general  event  debrief.  However,  a 
key  point  to  acknowledge  with  both  an  exercise,  as  well  as  a  real  event  debrief,  is  the  role 
that  stress  plays  in  these  types  of  meetings. 

The  reason  for  this  discussion  is  to  not  only  honor  the  effect  that  stress  has 
on  the  responders  but  to  also  take  time  to  consider  how  that  stress  may  affect  memory 
and  learning.  Ideally,  out-going  response  personnel  are  debriefed  after  every  operational 
shift.  However,  the  reality  is  that  after  the  end  of  an  exercise  or  actual  emergency,  all 
response  personnel  may  begin  to  feel  the  culminating  effects  of  the  entire  experience, 
including  elevated  levels  of  stress.  Butler,  Panzer,  and  Goldfrank  state  that: 

The  effect  of  exposure  to  a  traumatic  event  is  variable  and  specific  to  the 
individual;  both  psychological  and  physiological  responses  can  vary 
widely.  Social  context,  biological  and  genetic  makeup,  past  experiences, 
and  future  expectations  will  interact  with  characteristics  of  the  traumatic 
experience  to  produce  the  individual’s  psychological  response  (Ursano  et 
ah,  1992).  In  general,  those  exposed  to  a  traumatic  event  show  increased 
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rates  of  acute  stress  disorder,  posttraumatic  stress  disorder  (PTSD),  major 
depression,  panic  disorder,  generalized  anxiety  disorder,  and  substance  use 
disorder  (Kessler  et  al.,  1995). 135 

In  other  words,  one  individual’s  experience,  although  possibly  considered  less  stressful 
than  another’s,  is  somewhat  relative  based  on  multiple  factors  including  context. 

Much  of  the  AAR  literature  suggests  that  performing  an  event  debrief  as 
close  to  the  exercise  or  actual  event  is  preferred  and  ideal.  The  idea  is  that  the  closer  the 
review  is  to  the  actual  experience,  the  easier  it  will  be  to  remember.  However,  a  2005 
study  by  Kuhlman,  Piel,  and  Wolf  suggests  stress  does  have  an  adverse  affect  on 
memory.  The  authors  state,  “For  the  first  time,  we  show  an  impairing  effect  of 
psychosocial  stress  on  memory  retrieval  in  humans.”136  Thus,  this  traditional  practice 
may  need  to  be  reconsidered. 

To  clarify  even  further,  Oliver  Wolf,  in  a  2009  article  entitled  “Stress  and 
Memory  in  Humans:  Twelve  Years  of  Progress?,”  summarizes,  “In  a  series  of 
experiments  it  could  be  demonstrated  that  stress  or  cortisol  treatment  acutely  leads  to 
reduced  memory  retrieval  efficiency.  This  might  constitute  a  problem  during  exams 
(Schoofs  et  al.,  2008a)  or  during  testimonies.”137  In  many  ways,  the  AAR  acts  as  a  type 
of  testimony  as  to  the  events  encountered  within  the  exercise  or  real  emergency.  If 
memory  is  “reduced”  because  of  the  exercise  or  actual  event,  AAR  participants  may  think 
they  are  accurately  accessing  their  memories  when  in  fact  they  maybe  even  more 
susceptible  to  Taleb’s  narrative  fallacy. 

The  literature  is  scarce  on  how  stress  affects  the  collective  memory  of  a 
group.  However,  it  can  be  assumed  that  if  a  number  of  stressed  individuals  are  trying  to 
objectively  capture  and  synthesize  the  “what”  “how”  and  “why”  of  an  exercise  or  real 

133  Adrienne  S.  Butler,  Allison  M.  Panzer,  and  Lewis  R.  Goldfrank,  ed.  Chapter  2:  Understanding  the 
Psychological  Consequences  of  Traumatic  Events,  Disasters  and  Terrorism.  Preparing  for  the 
Psychological  Consequences  of  Terrorism:  A  Public  Health  Strategy’  (Washington,  DC:  The  National 
Academies  Press,  Institute  of  Medicine,  2003),  34-63. 

136  Sabrina  Kuhlman,  Marcel  Piel,  and  Oliver  T.  Wolf,  “Impaired  Memory  Retrieval  after 
Psychosocial  Stress  in  Healthy  Young  Men,  The  Journal  of  Neuroscience  25,  no.  1 1  (2005):  2979. 

137  Oliver  T.  Wolf,  “Stress  and  Memory  in  Humans:  Twelve  Years  of  Progress?”  Brain  Research 
1293  (2009):  151. 
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event,  their  various  abilities  to  access  stress-related  memories  may  prove  to  make  the 
charge  a  difficult  task.  This  type  of  situation  could  create  a  type  of  “Rashomon  effect”  in 
which  each  AAR  participant  is  recreating  the  experience  from  unique  stress-fueled 
perspectives  to  create  some  type  of  common  organizational  memory. 

Thus  far,  this  discussion  has  focused  on  stress  as  it  relates  to  memory 
within  the  context  of  the  AAR.  However,  as  mentioned  earlier,  Moynihan  discusses  the 
idea  of  individual  learning  during  the  crisis  itself.  In  this  sense,  the  question  now 
becomes,  “how  does  stress  affect  learning  and  memory”?  Schawbe  and  Wolf  address  this 
issue  in  their  article  entitled  “Learning  Under  Stress  Impairs  Memory  Formation.”  The 
author’s  state: 

In  summary,  we  show  that  learning  under  stress,  i.e.  during  the  early  phase 
of  the  stress  response,  can  have  detrimental  effects  on  subsequent  memory 
performance.  One  possible  explanation  seems  to  be  that  stress  acted  as  a 
distractor  during  encoding,  diverting  attention  from  the  learning 
materials.”138 

To  be  clear,  this  finding  states  that  trying  to  leam  under  stress  negatively  affects 
subsequent  memory  and  not  the  actual  ability  to  leam  or  problem  solve  in  real  time.  The 
research  cited  earlier  in  this  report  states  that  stress  negatively  affects  memory  in  general. 
Although  the  differences  may  be  slight,  both  findings  suggest  that  stress  can  impede  both 
learning  and  memory.  However,  even  if  stress  were  found  to  affect  real-time  learning  and 
problem  solving,  this  finding  would  support  the  use  of  RRF-type  configurations. 

In  light  of  the  data  presented,  the  idea  that  events  remembered  during 
stressful  situations  could  be  captured  in  any  type  of  “objective”  manner  would  be 
difficult  to  prove.  Ultimately,  the  researcher  believes  this  data  requires  that  the  practice  of 
closely  pairing  a  stressful  exercise  or  event  with  a  session  that  demands  the  use  of 
individual  or  collective  memory  to  create  “lessons  learned”  that  may  be  used  during  a 
subsequent  event  be  rethought.  Regardless,  as  suggested  by  the  subtext  of  this  thesis,  he 
would  submit  that  too  much  emphasis  is  placed  on  past  learning  and  memory  in 
relationship  to  organizational  learning  in  the  first  place. 

138  Lars  Schwabe  and  Oliver  T.  Wolf,  “Learning  Under  Stress  Impairs  Memory  Formation” 
Neurobiology  of  Learning  and  Memory,  no.  93  (2010):  188. 
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Regardless,  within  the  ELF,  the  AER  would  be  the  first  of  a  combination 
of  learning  and  planning  dialogues  that  follow  the  end  of  a  real  event  or  exercise.  These 
meetings  would  speak  to  the  implementation  of  various  intercrisis  learning  and  planning 
activities.  Thus,  the  AER  would  involve  “the  development  of  new  values  and  interests.  In 
this  case,  if  we  are  to  escape  the  icy  grip  of  technocratic  planning,  we  must  develop  a 
humanist  style  of  learning  through  planning  and  a  theory  of  planning  as  wide  spread 
social  learning.”139  In  this  respect,  the  AER  starts  the  process  of  intercrisis  learning  and 
planning  described  in  more  detail  within  the  next  section. 

b.  Conclusion 

The  negative  effect  of  stress  on  memory  and  learning  is  a  problem  not 
discussed  within  the  AAR  learning  literature.  A  literature  review  on  the  effects  of  stress 
on  memory  and  learning  suggests  that  debriefing  responders  as  a  way  to  capture  an 
accurate  account  of  events  or  as  a  way  to  discern  knowledge  and  learning  (immediately 
after  a  shift  or  the  actual  event)  may  not  be  a  wise  practice. 

Data,  information,  and  knowledge  captured  during  the  event  through  shift 
change  debriefings  or  via  tools,  such  as  the  RRF,  may  prove  to  be  more  accurate  and 
reliable  than  a  hot  wash  immediately  after  the  event.  Using  this  infonnation  in  subsequent 
and  ongoing  leaming/planning  meetings  (discussed  in  the  next  session),  as  opposed  to  the 
remembrances  of  stressed  out  responders,  should  provide  more  objective  and  rational 
data  for  later  analysis  and  evaluation. 

5.  Ongoing  Learning/Planning  Meetings 

a.  General  Discussion 

The  connection  between  learning  and  planning  cannot  be  overstated.  As  a 
reminder,  van  der  Heijden  stresses  that,  “In  a  situation  of  uncertainty  planning  becomes 
learning,  which  never  stops.”140  The  researcher  would  suggest  that  what  van  der  Heijden 
states  holds  true  not  only  during  an  emergency  but  after  as  well.  Thinking  that  if  the  time 

139  Bertram  Geross  in  Michael,  Learning  to  Plan  and  Planning  to  Learn,  346. 

140  van  der  Heijden,  Scenarios:  The  Art  of  Strategic  Conversation,  16. 
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in  between  events  is  used  to  just  update  the  plan  or  exercise,  the  capability  that  the 
perfect  response  can  be  ensured  is  fantasy.  In  this  respect,  planning  and  exercising 
becomes  just  another  way  to  gain  some  sort  of  imaginary  control  over  unordered  and 
uncertain  forces.  Of  course,  traditional  planning  methods,  such  as  learning  about  the  roles 
and  responsibilities  of  response  partners,  exercising  mutual  plans,  and  building 
relationships  are  still  (and  will  always  be)  necessary.  However,  these  types  of  activities 
need  to  be  complemented  formally  with  a  leaming/planning  capability  that  consists  of 
continually  surfacing  assumptions,  adapting  to  environmental  changes  (policy, 
environmental,  political,  etc.),  sensemaking,  systems  thinking,  tolerating  ambiguity,  etc. 

For  example,  using  these  on-going  leaming/planning  meetings  to  critically 
review  single  and  double  loop  learning  that  has  resulted  from  recent  exercises  or 
emergencies  would  offer  an  excellent  opportunity  to  delve  deeply  into  learning  that  is 
only  adaptive  to  a  specific  circumstances  (single  loop)  or  learning  that  is  more  generative 
in  nature  (double  loop),  and  has  the  ability  to  challenge  and  influence  the  values  and 
beliefs  of  the  organization. 141  This  type  of  “generative”  double  loop  learning  has  the 
potential  to  impact  the  highest  order  of  the  organization.  Ultimately,  keeping  the  single 
and  double  loop  conversation  at  the  forefront  of  these  types  of  meetings  is  an  excellent 
indirect  way  for  a  response  agency  to  engage  continually  in  an  ongoing  conversation 
about  organizational  values  and  beliefs  by  identifying  the  “gaps”  between  organizational 
forces  (assumptions,  beliefs,  etc.)  and  day-to-day  activities. 

The  problem  is  that  once  an  exercise  has  been  completed,  the  hot 
wash/AAR  has  been  conducted,  and  the  changes  have  been  made  to  the  plans  and 
procedures,  the  traditional  planning  model  is  essentially  complete  until  the  next 
exercise/event  cycle  has  begun.  As  planners,  it  is  possible  to  be  fooled  into  thinking  that 
until  this  next  cycle  begins,  immunity  to  the  complexities  of  the  next  emergency  is 
somehow  granted.  For  example,  the  researcher  recently  received  the  following  email 
from  an  esteemed  colleague  within  his  agency: 


van  der  Heijden,  Scenarios:  The  Art  of  Strategic  Conversation,  53. 
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The  following  is  a  link  to  the  Health  Alert  Network  where  you  can 
download  a  copy  of  the  latest  Go-Kit  (Go-Kit-23  August  2012.pdf)  that 
contains  plans,  SOPs,  guidance  and  other  information  necessary  for 
properly  managing  an  event  and  the  Agency  Operations  Center.  [Note:  the 
file  is  too  large  to  e-mail] 142 

In  essence,  the  message  of  this  email  is  that  now  that  the  latest  Go-Kit  has  been 
published,  nothing  more  is  necessary  to  “properly  manage  an  event.”  However,  the 
researcher  would  argue  that  the  subtext  of  this  communication  sends  a  false  message. 
How  can  any  one  set  of  documents  accurately  reflect  how  to  manage  a  future  event 
properly?  If  the  event  were  simple  or  complicated  (ordered),  it  is  possible  that  existing 
protocols,  procedures  and  capabilities  will  be  able  to  manage  the  crisis  effectively.  If  the 
event  were  complex  or  chaotic,  the  documents  in  the  Go-Kit  would  be  potentially  useless. 
This  example  is  similar  to  what  Donald  Michael  (in  “In  Search  of  the  Missing  Elephant: 
Selected  Essays”)  describes  when  he  suggests: 

One  of  the  functions  organizations  perform  is  to  buffer  the  individual 
member  from  the  impact  of  the  chaotic  interrelation  of  everything  to 
everything.  Ideally  organizations  free  the  member  to  deal  with  just  so 
much  of  the  environment  as  his  intellect  and  psyche  pennit. 143 

From  this  perspective,  the  email  has  an  underlying  message  that  suggests 
unordered  domains  do  not  exist.  The  logic  is  plans  based  upon  prior  events  will  be 
sufficient  to  help  manage  events  in  the  future.  Since  these  plans  will  be  all  that  is  needed, 
it  will  not  be  necessary  to  leam.  The  implication  is  the  Go-Kit  contains  all  the  answers. 
Graham  Leicester  states  it  as,  “In  other  words,  our  organizations  are  there  to  provide  a 
zone  of  competence — to  keep  the  ‘blooming,  buzzing  confusion’  [William  James,  1890, 
488]  outside  the  door  at  bay.  Don’s  [Donald  Michael]  point  is  that  the  confusion  is  too 
great  today  to  allow  that  approach.”144 

However,  the  value  of  ongoing  planning  and  learning  meetings  would  be 
to  explore  the  “chaotic  interrelation  of  everything  to  everything”  courageously.  The  goal 


142  Alan  Visnick,  e-mail  message  to  author,  August  21,  2012,  Portland,  OR. 

143  Donald  N.  Michael,  “In  Search  of  the  Missing  Elephant:  Selected  Essays,”  International  Futures 
Forum,  2010,  4. 

144  Ibid. 
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would  not  be  to  produce  answers  that  would  be  added  to  the  never-ending  barrage  of 
plans  and  procedures,  but  to  refine  and  redefine  the  questions  continually  and  embrace 
the  confusion.  As  Donald  Michael  states: 

Such  a  society  would  self-consciously  question  its  premises  as  well  as  its 
actions,  accept  uncertainty,  and  measure  leadership  competence  at  all 
levels  by  its  capacity  to  acknowledge  ignorance  and  uncertainty  as 
prerequisites  for  discovery  and  change,  i.e.  as  the  conditions  for 
learning. 145 

Ultimately,  the  work  of  these  ongoing  planning/learning  meetings  would  be  to  prepare 
for  the  ordered  domains  by  refining  and  updating  plans  and  procedures  while  also 
preparing  to  engage  the  unordered  domains  by  identifying  and  practicing  the  conditions 
for  learning.  These  meetings  would  also  allow  leaders  to  model  the  vulnerability  that 
must  precede  true  learning. 

b.  Conclusion 

The  problem  is  that  once  an  emergency  hot  wash/ AAR  has  occurred,  the 
AAR  has  been  written  and  the  plan  improvements  have  been  made,  an  underlying 
assumption  exists  that  planning  is  somehow  suspended  until  the  planning  cycle  begins 
again  with  the  next  exercise.  These  types  of  intercrisis  adjustments  can  potentially 
address  problems  in  the  ordered  domains  by  refining  plans,  procedures,  and  capabilities. 
However,  a  major  opportunity  is  being  missed  by  not  using  this  time  to  prepare  also  for 
complex  and  chaotic  events. 

In  this  context,  preparing  for  unordered  events  would  mean  cultivating  the 
organizational  conditions  for  learning  and  planning  in  real  time. 

6.  Intercrisis  (or  Prescriptive)  Learning 

Moynihan  describes  intercrisis  learning  as  “learning  from  one  crisis  and  making 
changes  to  prepare  for  another.”146  This  type  of  learning  is  normally  the  provenance  of 
the  AAR/hot  wash  and  any  recommendations  derived  from  the  subsequent  After  Action 

146  Michael,  “In  Search  of  the  Missing  Elephant:  Selected  Essays,”  17. 

146  Moynihan,  “Learning  under  Uncertainty:  Networks  in  Crisis  Management,”  352. 
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Report  and  improvement  plan.  For  example,  in  the  language  of  single  and  double  loop 
learning,  a  double  loop  recommendation  might  lead  to  the  implementation  of  a 
completely  new  response  system,  such  as  an  Incident  Command  System  (ICS).  However, 
a  single  loop  application  might  mean  that  an  agency  might  only  modify  certain  systems 
and  policies  over  time  instead  of  shifting  the  response  culture  with  one  major  policy 
change.147 

As  suggested,  the  AER  and  subsequent  planning  and  learning  meetings  could 
ideally  help  identify  medium  and  longer  term  single  and  double  loop  learning 
opportunities.  The  single  loop  learning  opportunities  might  take  the  shape  of  changes  to 
existing  policy  or  procedures.  Within  the  Cynelln  Framework,  these  types  of  adjustments 
would  influence  practice  that  exists  within  the  ordered  domains  (simple  and 
complicated).  For  example,  during  a  long-tenn  response,  multiple  Standard  Operating 
Procedures  (SOP)  may  be  drafted  (around  the  same  task),  which  indicates  an  ongoing 
amplification  or  dampening  process.  Thus,  when  this  particular  SOP  is  later  reviewed,  it 
can  be  detennined  if  the  practice  has  reached  a  predictable  cause  and  effect,  whereas  it 
can  be  placed  in  the  realm  of  best  practice,  or  if  the  procedure  is  still  somewhat  less 
predictable,  it  can  be  identified  as  good  practice.  Labeling  SOPs  as  best,  good  or  even 
emergent  will  help  planners  as  they  try  to  move  these  types  of  procedures  from  the 
unordered  to  ordered  domains. 

However,  before  response  agencies  can  begin  using  the  tools  and  concepts 
discussed  throughout  this  thesis,  they  must  first  be  educated  on  them.  This  type  of 
curriculum  would  complement  the  existing  training  curriculum  that  most  response 
agencies  currently  use.  For  example,  the  National  Incident  Management  System  (NIMS) 
requires  that  personnel  within  the  homeland  security  enterprise  have  certain  skills  and 
capabilities  (i.e.,  ICS  position  training,  hazardous  material  training).  For  the  most  part, 
these  types  of  competencies  are  training  centered,  which  means  they  are  essentially  task 
focused.  On  the  other  hand,  a  curriculum  designed  around  sensemaking,  complexity, 
managing  uncertainty,  etc.  would  need  to  be  more  education  focused.  In  other  words,  this 


147  Moynihan,  “From  tntercrisis  to  tntracrisis  Learning,”  197. 
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type  of  instruction  would  not  be  so  much  centered  on  “making  changes”  to  prepare  for 
the  next  crisis,  as  it  would  be  on  how  to  use  different  cognitive  tools,  frames,  and 
methods  to  help  adapt  and  leam  in  real  time  based  up  each  unique  circumstance.  Unlike 
the  intercrisis  learning  that  Moynihan  identifies,  this  type  of  learning  between  crises 
would  also  focus  on  developing  organizational  learning  capabilities.  Please  see  Appendix 
A  for  a  listing  of  courses  that  might  be  applied  to  this  type  of  curriculum. 
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V.  THE  CONCLUSION 


A.  DISCUSSION 

This  thesis  began  by  asserting  that  although  it  has  been  in  practice  for  almost  40 
years,  research  and  theory  to  support  the  AAR  are  few  and  the  approach  has  been 
difficult  to  validate  because  the  history  of  the  AAR’s  emergence  likely  did  not  allow  for  a 
traditional  grounding  in  theory  and  research.  Today,  to  continue  to  try  and  identity  one 
specific  foundational  model  for  a  process  now  being  practiced  in  a  multitude  of 
organizational  types  will  probably  not  influence  the  application  in  any  real  way. 

Even  if  various  writers  and  researchers  in  the  field  agreed  upon  a  theoretical 
model,  it  is  unlikely  that  the  practice  would  be  substantially  affected.  Moreover,  on  the 
flip  side,  if  it  was  detennined  that  the  efficacy  of  the  process  could  not  be  demonstrated, 
it  is  doubtful  that  organizations  would  stop  using  it. 

Ultimately,  the  real  problem  is  that  it  is  not  known  whether  the  use  of  the  AAR,  as 
currently  practiced,  leads  to  increased  individual  or  organizational  learning.  To  suggest 
that  something  has  been  learned  without  actually  “testing  the  lessons”  during  subsequent 
exercises  or  events  is  premature  and  misleading.  Likewise,  changing  a  plan  or  protocol 
based  on  one  unique  experience  and  suggesting  that  it  has  entered  the  realm  of  “best 
practice”  without  continually  testing  the  practice  in  a  multitude  of  settings  is 
inappropriate  and  possibly  negligent.  Moreover,  if  an  actual  lesson  (best  practice)  has 
been  generated,  it  is  difficult  to  capture  and  transfer  this  new  knowledge  effectively. 148 

In  the  end,  relying  on  “lessons  learned”  from  past  events  could  in  fact  leave 
emergency  responders  unable  truly  to  make  sense  of,  as  well  as  leam  from,  an  unordered 
event  occurring  in  real  time.  This  is  not  to  say  value  does  not  exist  in  examining  and 
observing  events  retrospectively  to  learn  both  from  failure  and  success.  However,  even  if 
learning  from  the  past  was  consistently  possible  and  could  be  effectively  shared,  is  it 
really  the  best  way  to  influence  practice  within  the  homeland  security  enterprise?,  which 

I48  Department  of  Homeland  Security,  Office  of  the  Inspector  General,  “FEMA’s  Progress  in 
Implementing  the  Remedial  Action  Management  Program.” 
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is  basically  the  real  topic  of  this  thesis:  how  can  agencies  within  the  homeland  security 
enterprise  create  the  individual  and  organizational  conditions  to  help  promote  reflection, 
action,  and  learning  both  during  and  after  exercises  and  events  to  improve  overall 
emergency  response? 

Regardless  of  the  benefits  of  canned  lessons  of  just-in-time  training,  responders 
need  to  also  be  prepared  for  just-in  time  learning,  for  it  is  learning  and  adapting  that  will 
prepare  everyone  for  the  future.  These  types  of  lessons  will  not  have  pre-determined  job 
action  sheets  or  lesson  plans  for  quick  reference;  but  will  be  the  unexpected  or  discovered 
lessons  experienced  and  learned  in  real  time.  Alternatively,  these  lessons  will  be  those 
that  have  their  genesis  during  a  crisis  but  may  not  truly  reveal  themselves  until  after  the 
emergency  has  ended.  As  Christopher  Bellavita  stresses,  “Our  stance  needs  to  shift  from 
the  desire  to  design  and  control  human  systems  (a  strategy  that  works  in  the  known  and 
knowable)  to  the  ability  to  recognize  and  influence  patterns  in  those  systems.  We  need  to 
learn  how  to  become  a  partner  with  an  uncontrollable  future.”149  This  stance  requires 
both  a  change  in  thinking,  as  well  as  a  change  in  tools  and  the  approach  to  learning. 

Partnering  with  an  uncontrollable  future  first  means  that  sense-making  tools  that 
can  distinguish  between  an  ordered  and  unordered  present  are  available  and  understood 
by  the  response  community.  Next,  it  is  essential  to  tolerate  the  ambiguity  that  an 
uncontrollable  future  will  certainly  bring.  After  that,  reflective  techniques  are  necessary 
that  will  make  it  possible  to  ask  the  right  questions  instead  of  focusing  on  finding  the 
right  answers.  Finally,  having  the  courage  to  act  in  the  face  of  uncertainty  and  continually 
shape  the  subsequent  results  must  be  a  part  of  the  equation.  These  components  form  the 
underlying  conditions  of  learning  and  fundamental  elements  of  the  ELF  developed  and 
presented  in  this  thesis. 

Real-time  learning  as  a  discrete  and  necessary  emergency  response  capability 
actually  fits  in  quite  well  with  the  federal  government’s  current  movement  towards  a 
larger  capabilities  (functions,  tasks,  etc.)  orientation.  To  try  to  implement  a  large-scale 
learning  organization  mentality  directly  within  the  homeland  security  enterprise  would  be 

149  Christopher  Bellavita,  “Changing  Homeland  Security:  Shape  Patterns,  Not  Programs,”  Homeland 
Security  Affairs  II,  no.  3  (October  2006):  12. 
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too  disruptive  and  never  implemented.  However,  if  a  learning  capability  was  prescribed 
as  an  addendum  to  existing  capabilities,  the  added  capability  might  indirectly  bring  about 
a  learning  organization  sensibility  within  the  homeland  security  enterprise. 

Thus,  trying  to  leverage  the  existing  system  (as  opposed  to  creating  a  new  one) 
using  a  similar  learning  language  (capabilities,  etc.)  might  be  the  best  way  to  influence 
the  larger  homeland  security  project.  Once  the  learning  capability  has  caught  on,  a  chance 
might  occur  that  the  idea  (learning  and  planning  in  real  time)  could  become  more 
substantial  as  it  is  woven  into  the  larger  enterprise  fabric.  The  tools  to  weave  this  tapestry 
exist  within  the  ELF. 

B.  FINAL  THOUGHTS 

Through  the  investigation  of  the  AAR  problem,  the  researcher  has  ultimately 
formulated  a  solution  related  to  the  larger  problem  of  short-  and  long-term  response 
agency  learning  and  planning.  For  him,  this  process  has  been  about  clarifying  his  vision 
for  organizational  development  within  a  response  agency.  The  process  has  produced  a 
framework  and  a  way  to  talk  about  crisis  learning  and  planning.  The  next  challenge  is  to 
implement  the  framework  as  a  whole  and  further  shape  it  as  necessary  for  relevant 
contexts. 

Through  this  process,  the  researcher  has  realized  only  a  few  concrete  answers 
actually  exist  within  the  emergency  management  game  that  can  successfully  address  each 
unique  crisis.  While  few  answers  exist,  it  is  very  important  to  engage  in  an  exploration  of 
useful  and  meaningful  questions  and  problems.  To  him,  the  ELF  helps  to  encourage  and 
promote  understanding  of  an  ongoing  strategic  conversation  and  helps  to  address  the  fact 
that  while  answers  are  few,  continual  reflection,  experimentation,  discovery,  and  learning 
will  help  craft  better  real-time  solutions. 

The  problem  is  that  a  method  to  address  honestly  the  fact  that  in  an  increasingly 
complex  world,  answers  to  homeland  security  questions  are  more  difficult  to  identify  and 
solve  than  ever  before,  does  not  exist.  Ultimately,  the  researcher  believes  that  ELF  solves 
this  problem.  By  framing  the  conversation  as  a  mixture  of  learning  and  planning,  he 
believes  it  might  be  possible  to  dodge  the  “plan  (or  capability)  as  an  answer  trap”  and 
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possibly  turn  response  agencies  into  organizations  that  move  from  making  plans  based  on 
the  past  to  having  the  confidence  to  learn  and  adapt  based  on  the  present  and  possible 
futures. 
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EPILOGUE 


Before  anyone  has  a  chance  to  answer  the  facilitator’s  question,  a  voice  rises 
among  the  crowd  and  asks: 

“Don’t  you  mean  the  AXR?  The  exercise  actions  are  over  and  so  is  the  exercise 
for  that  matter.  We  are  reviewing  the  exercise,  right?  Aren’t  we  doing  this  ELF  thing 
now?” 

For  a  second,  the  facilitator  looks  slightly  flustered  but  then  a  smile  crosses  her 

lips. 

“Yes,  thanks,  you  are  right... old  habits  die  hard.  So,  who  can  identify  an  example 
of  single  loop  learning  that  took  place  during  the  exercise?” 

“Well,  we  needed  to  make  some  changes  to  Ops  Center,  our  resource  ordering 
application.  Luckily  we  had  staff  from  state  emergency  management  here  with  us  so  that 
the  fix  could  be  done  in  real  time.  That  solved  the  problem.  But  I  think  this  quick  fix 
speaks  to  the  larger  issue  of  poor  collaboration.  I  know  we  always  say  that  we  value 
collaborating  with  our  partners.  But  if  this  problem  is  like  most  of  the  others,  it  will  just 
get  lost  in  the  paper  shuffle  and  nothing  will  happen.  More  than  this  being  an  Ops  Center 
issue,  I  see  this  as  a  challenge  to  our  organizational  values.  Do  we  truly  value 
collaboration  or  don’t  we?  If  we  do,  this  problem,  which  we  have  seen  before,  might  have 
been  solved  years  ago.” 

Once  again,  the  facilitator  appears  flustered;  however,  she  quietly  composes 

herself. 

“Wow,  you  certainly  took  the  recent  dialogue  and  mental  models  class  to  heart. 
You  are  right;  this  might  be  a  values  problem.  It  might  also  be  a  systems  problem  in  that 
we  will  be  unable  to  do  our  job  unless  our  Emergency  Management  partners  are  in 
alignment  with  our  understanding  of  the  larger  system.  Regardless,  you  have  definitely 
identified  a  double-loop  learning  opportunity.” 
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And  with  that,  most  in  the  room  peel  back  another  sticky  wrapper  and  prepare  for 
an  afternoon  of  surfacing  assumptions,  sensemaking,  identifying  single  and  double  loop 
learning,  tolerating  some  ambiguity  and  debating  whether  or  not  smart  practice  has 
entered  the  realm  of  best  practice. 

Now  we  are  learning  in  a  way  that  prepares  us  for  the  complexity  and  chaos  that 
the  2 1st  century  will  most  certainly  bring. 
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APPENDIX  A.  BUILDING  AN  ORGANIZATIONAL  LEARNING 

CAPABILITY 


A  core  template  or  model  for  this  type  of  curriculum  described  in  the  Intercrisis 
(or  Prescriptive)  Learning  section  can  be  found  in  Peter  Senge’s  The  Fifth  Discipline: 
The  Art  and  Practice  of  the  Learning  Organization.  In  The  Fifth  Discipline,  Senge 
discusses  what  he  calls  the  “core  learning  capabilities  for  teams.”150  These  learning 
capabilities  would  be  a  wonderful  capability  compliment  as  aspects  of  the  homeland 
security  enterprise  currently  move  to  a  more  capability  based  planning  model.  For 
example,  the  Centers  for  Disease  Control  (CDC)  have  “developed  15  capabilities  to  serve 
as  national  public  health  preparedness  standards.”151  Of  particular  interest  to  this  thesis, 
none  of  the  CDC  capabilities  addresses  the  ability  to  leam.  Thus,  Senge’s  “learning 
capabilities”  can  act  as  a  solid  addition  to  these  emerging  efforts. 

Senge  identifies  his  three  core  learning  capabilities  as  the  following. 

A.  ASPIRATION 

1.  Personal  Mastery 

Personal  mastery  is  the  discipline  of  continually  clarifying  and  deepening 
our  personal  vision,  of  focusing  our  energies,  of  developing  patience  and 
of  seeing  reality  objectively. 152 

2.  Shared  Vision 

The  practice  of  shared  vision  involves  the  skills  of  unearthing  shared 
‘pictures  of  the  future’  that  foster  genuine  commitment  and  enrollment 
rather  than  compliance. 153 


150  Peter  Senge,  The  Fifth  Discipline:  The  Art  and  Practice  of  the  Learning  Organization  (Doubleday, 
2006),  xvii. 

151  Centers  for  Disease  Control  and  Prevention,  Office  of  Public  Heath  Procedures  and  Response, 
“Public  Health  Preparedness  Capabilities:  National  Standards  for  State  and  Local  Planning,”  March  15, 
2012,  http://www.cdc.gov/phpr/capabilities/. 

152  Senge,  The  Fifth  Discipline:  The  Art  and  Practice  of  the  Learning  Organization,  7. 

153  Ibid.,  9. 
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B. 


REFLECTIVE  CONVERSATION 


1.  Mental  Models 

The  discipline  of  working  with  mental  models  starts  with  turning  the 
mirror  inward;  learning  to  unearth  our  internal  pictures  of  the  world,  to 
bring  them  to  the  surface  and  hold  them  rigorously  to  scrutiny. 154 

2.  Dialogue 

The  discipline  of  team  learning  starts  with  ‘dialogue,’  the  capacity  of 
members  of  a  team  to  suspend  assumptions  and  enter  into  a  genuine 
‘thinking  together.’ 155 

C.  UNDERSTANDING  COMPLEXITY 
1.  Systems  Thinking 

Systems  thinking  is  a  conceptual  framework,  a  body  of  knowledge  and 
tools  that  has  been  developed  over  the  last  fifty  years,  to  make  the  full 
patterns  clearer,  and  to  help  us  see  how  to  change  them  effectively. 156 

These  three  learning  capabilities  would  provide  a  powerful  accompaniment  to  the 
more  task  based  capabilities  currently  being  implemented.  In  addition  to  Senge’s  three 
core-learning  capabilities,  the  researcher  would  also  propose  courses  on  sensemaking  and 
tolerating  ambiguity  to  round  out  the  set. 

Ultimately,  the  problem  is  that  although  the  homeland  security  enterprise  is 
moving  towards  a  capability  based  planning  model,  it  is  missing  one  important  piece:  the 
capability  to  leam.  Although  HSEEP  provides  guidance  on  how  to  identify  gaps  and 
implement  changes,  it  does  not  build  an  individual  or  organizational  learning  capability. 
What  is  needed  is  specific  education  on  the  component  parts  of  organizational  learning 
(such  as  deciphering  between  single  and  double  loop  learning),  how  to  engage  in 
reflective  conversation,  the  ability  to  understand  complex  systems,  and  how  to  make 
sense  of  an  increasingly  complex  and  uncertain  world.  Complementing  the  existing 
homeland  security  capabilities  with  a  set  of  core  learning  capabilities  can  help  ensure  that 

l^4  Senge,  Tlte  Fifth  Discipline:  The  Art  and  Practice  of  the  Learning  Organization,  10. 

155  Ibid.,  7. 

156  Ibid. 
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when  plans,  procedures  and  pre-identified  capabilities  are  not  sufficient  to  the  task, 
individual  responders  will  have  a  set  of  foundational,  critical  thinking  skills  and 
capabilities  that  have  prepared  them  to  reflect  and  act  on  their  feet  to  help  them  manage 
the  ensuing  uncertainty  and  disorder. 
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APPENDIX  B.  VISUAL  REPRESENTATION  OF  THE  EMERGENCY  LEARNING  FRAMEWORK 


Intracrisis  Learning  Frames 

(Intercrisis)  Debrief  Frames 

Intercrisis 

Learning/Planning 

Meetings 

Intercrisis  Learning  (Post  event 
learning — Prescriptive  learning) 

Exercise 

After  Action  Review  (AAR) 

A  period  of  reflection,  action  and 
learning  injected  into  the  exercise  (not 
after). 

•  Facilitated  by  a  Learning  Unit 
Leader  (or  other  trained 
personnel) 

•  To  be  initiated  directly  after  a 
capability,  objective  or  any 
other  crucial  activity  has  been 
completed. 

Underlying  Theories/Concepts: 

Emergent  Learning,  Experiential 

Learning,  Adaptation,  Cynefin 

Framework,  Learning  Organization, 
Appreciative  Inquiry 

After  Exercise  Review  (AXR)  or 

Hot  Wash 

To  be  initiated  after  the  Exercise  has 
been  completed 

•  Identify  Cynefin  Learning 
Domains 

•  Identify  medium  and  longer 
tenn  learning  opportunities 

•  ID  single/double  loop 
learning  opportunities 

•  ID  new  issues  and  learning 
gaps 

•  Facilitated  by  a  Learning 

Unit  Leader 

Ongoing  Planning 
Meetings/  Exercises/ 
Formal  and  Informal 
Conversations/Education 
and  Training 

Ongoing  Training  and 

Education  (T  and  E) 

T  and  E  2.0  (proposed) 

•  Learning  Organization 
Course 

Systems 

Mental  Models 
Double/Single 

Loop  Learning 

•  Sensemaking  Course 
(Cynefin) 

•  Scenario  Planning 

•  Reflective  Practitioner 

•  Managing  Uncertainty 

•  Learning  How  to  Learn 

•  Surfacing  Assumptions 

•  Managing  Uncertainty 

•  Network  Theory 

•  Organizational  Culture 
(Vision,  values,  etc) 

T  and  E  1.0  (Current) 

•  Incident  Command 
Courses 

•  Hazardous  Materials 
Courses 

•  Online  FEMA 
Coursework 
(Independent  Study) 

•  Etc. 

Event 

During  Event  Reflection  (DER) 

To  be  initiated  during  every  operational 
shift  change  or  when  otherwise 
appropriate. 

•  Facilitated  by  a  Learning  Unit 
Leader  (or  other  trained 
personnel) 

•  Use  of  Rapid  Reflection  Force 
Underlying  Theories/Concepts: 

Emergent  Learning,  Experiential 
Learning,  Adaptation,  Cynefin, 
Single/Double  Loop,  Systems  Thinking, 

After  Event  Review  (AER)  or  Hot 
Wash 

To  be  initiated  after  emergency 
operations  have  been  fully  completed. 

•  Facilitated  by  a  Learning 

Unit  Leader 

•  Identify  medium  and  longer 
tenn  learning  opportunities 

•  ID  single/double  loop 
learning  opportunities 

•  ID  new  issues  and  gaps 
Facilitated  by  a  Learning 

Unit  Leader 
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